Crawl
AI-generatedSummary
Initiates a web crawl from a starting URL, optionally polling for completion to return content from each visited page.
Inputs
- url (required) — The starting URL to begin the crawl.
- maxDepth — Maximum link depth to crawl (defaults to 2).
- maxPages — Maximum number of pages to crawl (defaults to 100).
- crawlFormats — Output formats to return for each crawled page (defaults to Markdown).
- crawlOnlyMainContent — Whether to strip navigation, footer, and sidebar to return only main content (defaults to true).
- waitForCompletion — Whether to poll the API until the crawl finishes (defaults to true).
- pollInterval — Seconds between status checks (only used if waitForCompletion is true).
- maxWaitTime — Maximum seconds to wait for the crawl to complete (only used if waitForCompletion is true).
Output shape
One item per crawled page (or a single status object if the crawl fails or no pages are returned).
Each output item contains a JSON object representing a crawled page, including fields like url, title, content (in the requested format), and status.
Examples
Example 1: Crawl a website and return page content in Markdown format immediately.
Set waitForCompletion to false and specify URL, maxDepth, and crawlFormats.
Example 2: Wait for a crawl to finish and process each page sequentially.
Set waitForCompletion to true to poll until completion, limiting maxWaitTime to avoid hanging.