From 190056ee8d6a4eca61d92a79cc25aad645e69d4a Mon Sep 17 00:00:00 2001 From: Philipp Tanlak Date: Mon, 13 Nov 2023 22:36:15 +0100 Subject: Move docs to flyscrape.com (#11) --- docs/configuration/domain-filter.md | 44 ------------------------------------- 1 file changed, 44 deletions(-) delete mode 100644 docs/configuration/domain-filter.md (limited to 'docs/configuration/domain-filter.md') diff --git a/docs/configuration/domain-filter.md b/docs/configuration/domain-filter.md deleted file mode 100644 index e8adc30..0000000 --- a/docs/configuration/domain-filter.md +++ /dev/null @@ -1,44 +0,0 @@ -# Domain Filter - -The `allowedDomains` and `blockedDomains` config options allow you to specify a list of domains which are accessible or blocked during scraping. - -```javascript -export const options = { - url: "http://example.com/", - allowedDomains: ["subdomain.example.com"], - // ... -}; -``` - -### `allowedDomains` - -This config option controls which additional domains are allowed to be visted during scraping. The domain of the initial URL is always allowed. - -You can also allow all domains to be accessible by setting `allowedDomains` to `["*"]`. To then further restrict access, you can specify `blockedDomains`. - -Example: - -```javascript -export const options = { - url: "http://example.com/", - allowedDomains: ["*"], - // ... -}; -``` - -### `blockedDomains` - -This config option controls which additional domains are blocked from being accessed. By default all domains other than the domain of the initial URL or those specified in `allowedDomains` are blocked. - -You can best use `blockedDomains` in conjunction with `allowedDomains: ["*"]`, allowing the scraping process to access all domains except what's specified in `blockedDomains`. - -Example: - -```javascript -export const options = { - url: "http://example.com/", - allowedDomains: ["*"], - blockedDomains: ["google.com", "bing.com"], - // ... -}; -``` -- cgit v1.2.3