diff options
Diffstat (limited to 'docs/configuration/url-filter.md')
| -rw-r--r-- | docs/configuration/url-filter.md | 42 |
1 files changed, 0 insertions, 42 deletions
diff --git a/docs/configuration/url-filter.md b/docs/configuration/url-filter.md deleted file mode 100644 index e2feda8..0000000 --- a/docs/configuration/url-filter.md +++ /dev/null @@ -1,42 +0,0 @@ -# URL Filter - -The `allowedURLs` and `blockedURLs` config options allow you to specify a list of URL patterns (in form of regular expressions) which are accessible or blocked during scraping. - -```javascript -export const options = { - url: "http://example.com/", - allowedURLs: ["/articles/.*", "/authors/.*"], - blockedURLs: ["/authors/admin"], - // ... -}; -``` - -### `allowedURLs` - -This config option controls which URLs are allowed to be visted during scraping. When no value is provided all URLs are allowed to be visited if not otherwise blocked. - -When a list of URL patterns is provided, only URLs matching one or more of these patterns are allowed to be visted. - -Example: - -```javascript -export const options = { - url: "http://example.com/", - allowedURLs: ["/products/"], -}; -``` - -### `blockedURLs` - -This config option controls which URLs are blocked from being visted during scraping. - -When a list of URL patterns is provided, URLs matching one or more of these patterns are blocked from to be visted. - -Example: - -```javascript -export const options = { - url: "http://example.com/", - blockedURLs: ["/restricted"], -}; -``` |