Crawl4AI Plus Advanced icon

Crawl4AI Plus Advanced

Advanced web crawling and extraction with full Crawl4AI API control

Actions15

Crawl Multiple URLs

AI-generated

Summary

Crawls multiple pages either from an explicit URL list or by discovering links from a seed URL, then returns formatted crawl results. Configure browser behavior, crawl limits and filters, content extraction, and output fields in the corresponding option groups.

Inputs

  • Crawl Mode — Choose Manual URL List (default) or Discover From Seed URL.
  • URLs (required) — In manual mode, provide URLs one per line or comma-separated. Optional denylist paths filter matching URLs before the API call; blocked URLs are reported in _safetyFilter on the first output item.
  • Seed URL (required) — In discovery mode, the starting URL from which the crawler follows links.
  • Discovery Query — Keywords used to score links for Best-First discovery; the UI says this query is required for that strategy.
  • Discovery Strategy — In discovery mode, choose Best-First, BFS (default), or DFS. Options set crawl depth, page and per-page link limits, external-domain handling, include/exclude URL patterns and domains, prefetch, result limit, and optional JSON resume state. Denylist paths are merged into the server-side URL pattern filter.
  • Browser & Session — Shared browser and session settings, including browser profile and engine, headers, cookies and storage state, JavaScript, proxy, timeout, viewport, and persistent session options.
  • Crawl Settings — Shared crawl behavior such as cache mode, robots.txt checking, content selectors and exclusions, JavaScript to run, wait conditions, retries, and anti-bot settings.
  • Output & Filtering — Choose markdown output, optional HTML, links, media, tables, screenshots, PDF, and SSL certificate data. Configure content filtering and table extraction; LLM-based options require LLM credentials and their applicable instruction or query.

Output shape

A list of crawl result items, generally one per crawled page.

Results use the node's structured crawl-result format. In manual mode, denylist-filtered URLs are summarized in _safetyFilter on the first item; the number of returned results can be limited in discovery mode.

Links

Discussion