H2 Crawl and Scrape icon

H2 Crawl and Scrape

Crawl websites and extract data (Cheerio, Puppeteer, Playwright)

H2 Crawl and Scrape

AI-generated

Summary

Crawl websites starting from one or more URLs using a selected crawler (Cheerio, Puppeteer, or Playwright) to extract either links, text content, or HTML content, with configurable crawling behavior.

Inputs

  • Crawler Type (required) — Choose the crawling engine: Cheerio (headless), Puppeteer (browser), or Playwright (browser).
  • Start URLs (required) — Comma-separated list of URLs to begin crawling from.
  • Append Timestamp — Optionally append a timestamp parameter to each URL to prevent caching.
  • Randomize User Agent — Randomize the User-Agent header for each request from a provided list or use the first if disabled.
  • User Agents (JSON Array) — A JSON array of User-Agent strings to use when randomizing or setting User-Agent headers.
  • Custom Headers (JSON) — Custom HTTP headers as a JSON object to include with each request.
  • Cookies (JSON Array of {Name, Value, Domain?}) — Cookies to set for requests, applicable only with Puppeteer or Playwright crawlers; ignored by Cheerio.
  • Use Proxy — Enable proxy use for crawling requests.
  • Proxy List (Comma-Separated) — Comma-separated list of proxy URLs to cycle through if proxy usage is enabled.
  • Session Management — Enable managed sessions to maintain cookies and state across requests.
  • Request Concurrency — Number of parallel requests to perform during crawling.
  • Delay Between Requests (Ms) — Base delay in milliseconds between requests.
  • Jitter Max (Ms) — Maximum random additional delay in milliseconds added to base delay to avoid detection.
  • Retry Count — Number of retry attempts for failed requests.
  • Crawl Depth — Maximum depth for following links detected on pages (0 means no following).
  • Follow Links — Enable following and crawling links found on each crawled page.
  • Link Pattern (Regex) — Regular expression to filter which links are followed. If empty, all links are followed.
  • Max Links Per Page — Maximum number of links to enqueue for crawling per page when following links.
  • Max Requests Per Crawl — Limit on total number of requests in the crawl to avoid infinite crawling.
  • Stop Condition (JS Expression Using crawledData) — JavaScript expression evaluated on crawled data length to stop crawling early, e.g., crawledData.length > 10.
  • Output Type (required) — Type of data to extract from pages: links (URLs), text content, or raw HTML.
  • Max Content Length (Chars) — Maximum length of extracted text or HTML content captured per page to prevent excessive memory use.

Output shape

a single JSON object with keys depending on outputType: an array of extracted unique links, texts, or HTML blocks, plus optional errors encountered during crawling

The output includes {links} if extracting links, {texts} if extracting text, or {htmls} if extracting HTML, each being arrays of unique strings. If any requests failed, an {errors} array details failed URLs and error messages. Crawling respects the configured max requests, depth, concurrency, and stop condition. If stop condition is invalid or unsafe, it is ignored. Output is a single item containing these results.

Discussion