CRW

Scrape, crawl, and extract web data with CRW — the open-source web scraper built for AI agents

Actions6

Check Crawl Status

AI-generated

Summary

Polls the status and retrieves data for an existing CRW crawl job.

Inputs

  • crawlJobId (required) — The ID of the crawl job to check.

Output shape

One item per page discovered, plus metadata fields for the job.

If the job is still running or incomplete, returns the job status and progress metadata. If data exists, returns each crawled page as a separate item with status, total, and completed counts.

Examples

Example 1: Check the progress of a crawl started in a previous step.

Enter the crawlJobId returned by a prior Crawl operation.

Example 2: Retrieve the final scraped content once a crawl completes.

Enter the crawlJobId; returns a list of items, one per page.

Links

Discussion