Website Crawler icon

Website Crawler

Fast website crawler with sitemap discovery, metadata extraction, and link mapping. By John Rippy.

Website Crawler

AI-generated

Summary

Crawls pages starting from the supplied URLs using the Apify Website Crawler actor. For each input item, returns the crawl status and either page results with a status-code summary or a message if the run did not succeed.

Inputs

  • Start URLs (required) — Comma-separated URLs to begin crawling; whitespace and empty entries are removed.
  • Max Pages — Maximum pages to crawl; defaults to 25. The UI describes a range of 1–500.
  • Max Crawl Depth — Maximum link depth to follow; defaults to 2. The UI describes a range of 0–10.
  • Crawl Sitemap — Whether to use sitemap discovery; defaults to true.
  • Timeout (seconds) — How long to wait for the Apify run to finish; defaults to 300 seconds.

Output shape

One output item per input item.

A successful run returns runId, status, summary (totalPages, successfulPages, errorPages, and redirectPages), and pages from the Apify dataset. A run with another status returns runId, status, and a message. Requires Apify API credentials.

Links

Discussion