AnakinInstall

Summary

Crawls a website starting from a specified URL, fetching multiple pages up to a configurable depth and page limit, optionally using a headless browser, and applying URL inclusion or exclusion patterns.

Inputs

• crawlUrl (required) — The starting URL to begin crawling from.
• crawlOptions — A collection of optional parameters to control crawling behavior, including:
• Country Code: two-letter proxy egress country code.
• Depth: number of link-hops to follow from the starting URL (default 1).
• Exclude Patterns: comma-separated URL patterns to skip.
• Include Patterns: comma-separated URL patterns to restrict fetched URLs.
• Max Pages: maximum number of pages to fetch (default 10).
• Max Wait Time: max time in seconds to wait for the crawl job to finish (default 300).
• Poll Interval: seconds between status checks (default 3).
• Session ID and Session Name: optional saved browser session identifiers for login-protected sites.
• Use Browser: whether to render pages with a headless browser (useful for SPA).

Output shape

a single JSON object containing the crawl job result including success status, job ID, original URL, and details of crawled pages.

The operation submits a crawl job to the Anakin API and polls for completion until the crawl finishes, fails, or the max wait time is exceeded. If the crawl completes successfully, the full crawl data is returned. Errors during crawling or timeouts are thrown as exceptions. Optional fields allow filtering URLs and configuring crawl depth and breadth.

Examples

Example 1: Crawl a site starting from https://example\.com with default options

Set 'URL' to 'https://example.com' and leave 'Additional Options' unset or default.

Links

• n8n community nodes installation guide