CRW

Scrape, crawl, and extract web data with CRW — the open-source web scraper built for AI agents

Actions6

Map

AI-generated

Summary

Discovers all URLs on a website up to a specified link depth, optionally using the site's sitemap.

Inputs

  • url (required) — The URL to discover links from.
  • mapMaxDepth — Maximum link depth to discover. Defaults to 2.
  • useSitemap — Whether to use the sitemap to discover URLs. Defaults to true.

Output shape

A list of items, one for each discovered URL.

If no links are found, returns the raw response object containing status and links.

Examples

Example 1: Map all links on a homepage up to depth 3 without using a sitemap.

URL: https://example.com, Max Depth: 3, Use Sitemap: false

Example 2: Discover links from a blog's sitemap, limiting depth to 1.

URL: https://blog.example.com, Max Depth: 1, Use Sitemap: true

Links

Discussion