AnakinInstall

Summary

Crawl a website starting from the specified URL, fetching markdown content across multiple linked pages based on configurable options such as crawl depth, page limits, inclusion/exclusion patterns, and browser rendering.

Inputs

• crawlUrl (required) — The starting URL to begin crawling from.
• crawlOptions — Additional optional parameters including: Country Code for proxy egress, Depth for the number of link-hops to follow, Exclude Patterns to skip matching URLs, Include Patterns to fetch only matching URLs, Max Pages as a hard page fetch limit, Max Wait Time (seconds) to wait for job completion, Poll Interval (seconds) between status checks, optional Session ID and Session Name for login-protected sites, and Use Browser to render pages in a headless browser (for SPAs).

Output shape

a single JSON object containing the crawl job result, including success status, job ID, the original URL, and details returned from the API about the crawl.

The operation submits a crawl job to the API and polls until completion or timeout. It returns detailed crawl results or error information. The output includes status fields and fetched content data as provided by the API.

Examples

Example 1: Crawl a website's pages up to 2 link-depth and a maximum of 20 pages, including only URLs matching certain patterns, rendered via browser

Set 'URL' to the starting point, 'crawlOptions.depth' to 2, 'crawlOptions.maxPages' to 20, specify 'crawlOptions.includePatterns' with desired URL patterns, and enable 'crawlOptions.useBrowser'.

Links

• n8n community nodes installation guide