summaryrefslogtreecommitdiff
path: root/docs/configuration/url-filter.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/configuration/url-filter.md')
-rw-r--r--docs/configuration/url-filter.md42
1 files changed, 0 insertions, 42 deletions
diff --git a/docs/configuration/url-filter.md b/docs/configuration/url-filter.md
deleted file mode 100644
index e2feda8..0000000
--- a/docs/configuration/url-filter.md
+++ /dev/null
@@ -1,42 +0,0 @@
-# URL Filter
-
-The `allowedURLs` and `blockedURLs` config options allow you to specify a list of URL patterns (in form of regular expressions) which are accessible or blocked during scraping.
-
-```javascript
-export const options = {
- url: "http://example.com/",
- allowedURLs: ["/articles/.*", "/authors/.*"],
- blockedURLs: ["/authors/admin"],
- // ...
-};
-```
-
-### `allowedURLs`
-
-This config option controls which URLs are allowed to be visted during scraping. When no value is provided all URLs are allowed to be visited if not otherwise blocked.
-
-When a list of URL patterns is provided, only URLs matching one or more of these patterns are allowed to be visted.
-
-Example:
-
-```javascript
-export const options = {
- url: "http://example.com/",
- allowedURLs: ["/products/"],
-};
-```
-
-### `blockedURLs`
-
-This config option controls which URLs are blocked from being visted during scraping.
-
-When a list of URL patterns is provided, URLs matching one or more of these patterns are blocked from to be visted.
-
-Example:
-
-```javascript
-export const options = {
- url: "http://example.com/",
- blockedURLs: ["/restricted"],
-};
-```