Tavily icon

Tavily

Search the web, extract page content, crawl and map websites, and run in-depth research with Tavily

Actions6

Crawl → URL

AI-generated

Summary

Crawl a website starting from the required root URL by sending it and the selected options to Tavily’s crawl endpoint. Use the options to control crawl scope, depth, result limit, extracted content, and response details.

Inputs

  • URL (required) — Root URL where the crawl begins.
  • Options — Optional crawl settings: allow following external-domain links; select or exclude domains and paths with regex patterns; set maximum links per level, crawl depth, and result limit; choose basic or advanced extraction and Markdown or text output; include images, favicons, or usage information; and provide natural-language instructions. Chunks Per Source is available when Instructions is set and accepts 1–5 chunks, each up to 500 characters.

Output shape

A single n8n item containing Tavily’s crawl response as JSON.

The node makes one POST request to Tavily’s /crawl endpoint per input item. Empty domain and path pattern lists are omitted from the request; populated lists are split on newlines and trimmed.

Links

Discussion