Spider icon

Spider

Crawl and scrape websites using the Spider API

Actions7

Crawl

AI-generated

Summary

Crawl a website starting from the given URL to extract content from multiple pages up to a specified page limit and optional depth.

Inputs

  • URL (required) — The starting URL to crawl
  • Page Limit — Maximum number of pages to return, minimum 1, default 50
  • Return Format — Format of the extracted content, one of Markdown, HTML, Plain Text, or XML
  • Output Mode — How to return multi-page results: as separate items or all pages in a single item
  • Additional Fields — Optional settings including anti-bot bypass, crawl depth, proxy usage, headless browser rendering, fetching full page content, including metadata, and applying readability processing

Output shape

A list of crawled pages as separate n8n items if 'One Item per Page' is selected, or a single item with an array of pages if 'All Pages in One Item' is selected.

Each returned page includes the extracted content in the selected format along with any requested metadata. The operation may follow links up to the specified crawl depth starting at the URL and stops once the page limit is reached.

Links

Discussion