Crawl4AI Plus Advanced icon

Crawl4AI Plus Advanced

Advanced web crawling and extraction with full Crawl4AI API control

Actions15

Crawl Multiple URLs

AI-generated

Summary

Crawl multiple URLs using either a manual list or a discovery-based approach from a seed URL, with extensive configuration options for crawl behavior, browser session, content extraction, content filtering, and output formatting.

Inputs

  • crawlMode (required) — Choose between 'Manual URL List' to provide explicit URLs or 'Discover From Seed URL' to recursively follow links starting from a seed URL.
  • urls (required) — For 'Manual URL List' mode, specify one or more URLs to crawl, separated by newlines or commas.
  • seedUrl (required) — For 'Discover From Seed URL' mode, specify the starting URL from which to begin link discovery and crawling.
  • query — Keywords guiding which links to follow during discovery; required when using Best-First discovery strategy.
  • discoveryStrategy — Options to customize the discovery crawl strategy (Best-First, BFS, or DFS) and set limits and filters on included or excluded domains, patterns, maximum depth, and more.
  • browserSession — Settings to configure the browser engine (Chromium, Firefox, WebKit), JavaScript enablement, stealth mode, custom cookies, headers, proxy, viewport, and session management for crawling.
  • crawlSettings — Advanced crawl settings such as anti-bot features, cache mode, respecting robots.txt, CSS selectors for content extraction, delays, retries, and link exclusion rules.
  • outputFiltering — Options to control content filtering strategy (BM25, LLM, pruning, none), markdown output format, inclusion of screenshots, PDFs, HTML, links, media, tables, and filtering thresholds.

Output shape

a list of crawl results, each including extracted content blocks, metadata, optionally screenshots and PDFs, and discovered links and media according to configured options

The output shape is a structured JSON array, containing detailed crawl results per URL. Content filtering and table extraction strategies affect returned data. Large result sets can be limited by 'Result Limit'. Verbose debug info can optionally be included.

Links

Discussion