Actions2
- Scrape Actions
Crawl
AI-generatedSummary
Manage web crawling tasks with operations to start, get status, cancel, resume, export results, or list crawls.
Inputs
- Operation (required) — Choose the crawl action to perform: start a new crawl, get crawl status, cancel a crawl, resume a crawl with retry options, export crawl results, or list crawls.
- URL (required) — The starting URL for a new crawl (required for Start Crawl operation).
- Crawl ID (required) — The identifier of an existing crawl, returned from starting a crawl, used for status, cancel, resume, and export operations.
- Resume Options — For resuming a crawl: choose whether to retry failed and/or cancelled pages and optionally add a comma-separated list of additional URLs to crawl.
- Export Options — For exporting crawl results: select output format (JSON, JSONL, CSV), filter pages by status, and specify which fields to include.
- List Filters — For listing crawls: apply filters such as limit, offset, crawl status, and domain substring to narrow results.
- Crawl Settings — For starting a crawl: configure crawling parameters including max pages, depth, output formats, URL include/exclude patterns, sitemap handling, JavaScript rendering mode, proxy usage, concurrency, delay, timeout, headers, extraction options, robots.txt respect, subdomain inclusion, change tracking, webhooks, screenshots, polling timeout, and more as detailed in the node parameters.
Output shape
One or more output items representing the crawl result or metadata, structure varies by operation. For example, starting a crawl returns crawl metadata including the crawl ID; status returns current crawl status and results; export returns parsed crawl pages as items; list returns a list of crawl summaries.
Start operation waits and polls until the crawl completes (with configurable timeout). Export operation returns crawl results parsed into individual n8n items per page. Cancel and resume operations perform actions without returning large datasets. List operation supports pagination and filtered retrieval of crawls.
Examples
Example 1: Start Crawl
Set the start URL and optionally configure crawl settings such as max pages, depth, extraction parameters, and wait for completion. The node returns crawl ID and results upon completion.
Example 2: Get Status
Provide a Crawl ID to query the current status and partial or full results of that crawl.
Example 3: Cancel Crawl
Provide a Crawl ID to cancel an active crawl, which refunds unused credits.