AnakinInstall

Summary

Crawl a website starting from a specified URL to bulk-fetch markdown content across multiple pages, following links up to a specified depth and within optional include/exclude URL patterns.

Inputs

• URL (required) — The starting URL to crawl.
• Additional Options — Optional parameters to control crawl behavior, including country code for proxy egress (default 'us'), crawl depth (link-hops from the starting URL, default 1), exclude patterns (comma-separated glob/regex to skip URLs), include patterns (comma-separated glob/regex to include only matching URLs), maximum pages to fetch (default 10), maximum wait time in seconds to complete crawl (default 300), polling interval in seconds for status checks (default 3), optional session ID and name for login-protected sites, and whether to render pages using a headless browser (default false).

Output shape

a single record containing the crawl job status and results upon completion

The operation initiates an asynchronous crawl job and polls until completion or timeout based on the max wait time. If successful, it returns a JSON object including success status, operation name, job ID, the original URL, and crawl results. In case of failure or timeout, an error is raised.

Links

• n8n community nodes installation guide