Crawl Site
AI-generatedSummary
Crawl multiple pages from a specified website URL, extracting selected data fields from each page with configurable crawling options.
Inputs
- URL (required) — The starting URL of the website to crawl.
- Crawl Output Fields — Select which data fields to extract from each crawled page, such as full HTML, links, media, meta tags, structured data, and text content.
- Crawl Options — Options to configure crawling behavior, including the maximum number of pages to crawl (1-100) and the number of parallel extraction workers (1-10).
Output shape
a single JSON object containing the original URL, an array of crawled pages with their extracted data, a count of pages crawled, and summary metrics such as total time taken and credits used
The operation performs a POST request to the DataGait API crawl endpoint and processes an SSE stream. The output JSON includes a 'pages' array with individual page data objects corresponding to the selected output fields. It may also include 'total_ms', 'total_credits_used', and a 'credit_exhausted' flag if credit limits are hit. Errors related to quota, rate limits, or concurrency are thrown with descriptive messages.