diff options
| author | Philipp Tanlak <philipp.tanlak@gmail.com> | 2023-10-17 18:16:58 +0200 |
|---|---|---|
| committer | Philipp Tanlak <philipp.tanlak@gmail.com> | 2023-10-17 18:16:58 +0200 |
| commit | 11d73f57a80bb65b7507ec80433b8f035ed226c2 (patch) | |
| tree | 6613f9f58d4a91f7267f852e910b5b73015f4db5 /README.md | |
| parent | 36846200df6e53723149b5f23ae78913290f6f8d (diff) | |
Allow configuration of links to follow
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 1 |
1 files changed, 1 insertions, 0 deletions
@@ -140,6 +140,7 @@ Below is an example scraping script that showcases the capabilities of flyscrape export const config = { url: "https://example.com/", // Specify the URL to start scraping from. depth: 0, // Specify how deep links should be followed. (default = 0, no follow) + follow: [], // Speficy the css selectors to follow (default = ["a[href]"]) allowedDomains: [], // Specify the allowed domains. ['*'] for all. (default = domain from url) blockedDomains: [], // Specify the blocked domains. (default = none) allowedURLs: [], // Specify the allowed URLs as regex. (default = all allowed) |