AnakinInstall

Summary

Crawl a website starting from a specified URL to bulk-fetch markdown content across multiple pages, following links up to a specified depth and applying optional include/exclude URL patterns.

Inputs

• URL (required) — The starting URL to begin the crawl
• Country Code — Two-letter code specifying the proxy egress country, defaulting to 'us'
• Depth — Number of link-hops from the starting URL to follow, default is 1
• Exclude Patterns — Comma-separated glob or regex patterns; any matching URLs are skipped
• Include Patterns — Comma-separated glob or regex patterns; only URLs matching these patterns are fetched
• Max Pages — Maximum number of pages to fetch, default is 10
• Max Wait Time (Seconds) — Maximum time to wait for the crawl job to complete, default is 300 seconds
• Poll Interval (Seconds) — Interval between status checks for the crawl job, default is 3 seconds
• Session ID — Optional saved browser session ID for login-protected sites
• Session Name — Optional saved browser session name
• Use Browser — Flag to render pages in a headless browser for single-page applications, default is false

Output shape

a single record containing crawl job status and results

The operation submits a crawl job to the Anakin API, then polls the job status until completion or timeout. Returns a JSON object with success status, the original URL, job ID, and fetched content or error details. Errors during crawling or timeouts are reported as node operation errors.

Examples

Example 1: Initiate a crawl starting at 'https://example\.com' with depth 2 and maximum 20 pages

Set URL to 'https://example.com', Depth to 2, Max Pages to 20, and optionally adjust other parameters like Use Browser or Country Code.

Links

• n8n community nodes installation guide