AnakinInstall

Summary

Crawl a website starting from the specified URL to bulk-fetch markdown content across multiple linked pages, optionally using a headless browser and with configurable crawling depth, page limits, URL inclusion/exclusion patterns, and session parameters for authenticated sites.

Inputs

• crawlUrl (required) — The starting URL to crawl
• crawlOptions — Collection of optional crawling parameters including: Country Code (two-letter proxy egress country code), Depth (link-hops from starting URL to follow), Exclude Patterns (comma-separated glob/regex patterns to skip URLs), Include Patterns (comma-separated glob/regex patterns to include URLs), Max Pages (hard cap on pages fetched), Max Wait Time (maximum seconds to wait for crawl completion), Poll Interval (seconds between status checks), Session ID (optional saved-browser-session ID for login-protected sites), Session Name (optional saved-browser-session name), and Use Browser (whether to render pages in a headless browser for SPAs).

Output shape

a single JSON object containing the crawl job results including status and fetched content details

The operation submits a crawl job to the API and polls for completion up to the Max Wait Time. If successful, it returns a JSON object with the job ID, requested URL, status, and crawl results. Errors in the crawl job or timeouts cause operation errors.

Examples

Example 1: Crawl starting from a homepage with default options

Set 'crawlUrl' to the homepage URL and use default 'crawlOptions' to crawl 1 level deep up to 10 pages without browser rendering.

Links

• n8n community nodes installation guide