Actions9
- Extract Actions
- Bulk Job Actions
- Crawl Job Actions
- Search Actions
- Account Actions
Crawl Job → Submit and Wait
AI-generatedSummary
Submit a crawl starting at the required Root URL, then poll until the job reports done and return its page results as separate n8n items.
Inputs
- Root URL (required) — URL where the crawl starts.
- Depth — Maximum breadth-first depth from the root URL; defaults to 1. Plan limits may cap the requested depth.
- Render JavaScript — Fetch pages through Playwright to capture JavaScript-rendered content; defaults to false and can make each page slower.
- Output Format — Representation for each page: Markdown (default), JSON Blocks, Chunks, Raw HTML, or Links. JSON Blocks and Chunks require Pro or higher.
- Retry — Server-side retry attempts per page after a timeout; defaults to 0, meaning one attempt.
- Max Pages — Optional successful-page limit that can narrow the plan cap. Defaults to 0, which leaves the plan cap in effect.
- Compliance Overrides — Optionally restrict allowed domains, add deny globs, or set robots handling. Overrides can narrow the API key policy only; unset fields are omitted.
- Poll Interval (Seconds) — How often to check job status; defaults to 2 seconds.
- Timeout (Seconds) — Maximum wait time; defaults to 300 seconds. The node throws if the job is not done by then.
Output shape
A list of n8n items, one per crawl result page. Each item includes the job ID, job kind, and that page's result fields.
If the completed job has no results, returns one item containing the job envelope. Each output item is paired with its input item. Configure Submit plus Get Status if the job may exceed the timeout or n8n execution budget.
Examples
Example 1: Submit a documentation site crawl and collect its pages before downstream processing.
Set Root URL to the site's starting page. Adjust Depth, Max Pages, rendering, format, retries, and compliance overrides as needed; tune polling and timeout for the expected crawl duration.