Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Scraping → Batch scrape multiple URLs simultaneously

AI-generated

Overview

This node performs batch scraping of multiple URLs simultaneously using the Firecrawl API. It is designed to extract content from multiple web pages in parallel, applying the same scraping settings to each URL. This is useful for scenarios where you need to gather data from many pages quickly, such as monitoring multiple product pages, aggregating news articles, or collecting research data from various sources.

Use Case Examples

  1. Scraping multiple product pages from an e-commerce site to compare prices and availability.
  2. Batch extracting article summaries from news websites for content aggregation.
  3. Collecting structured data from multiple competitor websites for market analysis.

Properties

Name Meaning
URLs The URLs to scrape in batch. Supports single URL, multiple URLs separated by commas or newlines, JSON array, or array expression from a previous node. Each URL is scraped independently with the same settings.
Parsers File parsers to enable for content extraction, such as extracting PDF content as markdown.
Scrape Options Configuration for how content is extracted from pages, including output formats, content filtering, timing, caching, and browser emulation settings.
Additional Fields Additional custom JSON properties to add to the request body.
Use Custom Body Whether to use a custom request body instead of the default structured input.
Request Options Node-level request options such as batching, SSL handling, proxy, and timeout settings.

Output

JSON

  • url - The URL that was scraped.
  • content - The extracted content from the page in the requested format(s).
  • status - The status of the scraping operation for each URL (e.g., success, failure).
  • metadata - Additional metadata about the scraping result, such as timestamps or error details.

Dependencies

  • Firecrawl API key credential

Troubleshooting

  • Invalid URLs in the input may cause the entire batch to fail if 'Ignore Invalid URLs' is disabled. Ensure URLs are correctly formatted.
  • Timeout errors can occur if pages take too long to load; increase the 'Timeout (Ms)' setting if needed.
  • SSL certificate errors may happen if the target site uses self-signed or expired certificates; enable 'Skip TLS Verification' cautiously.
  • Rate limiting may occur if 'Max Concurrency' is set too high; reduce concurrency to avoid throttling.
  • Webhook configuration errors can prevent progress notifications; verify webhook URLs and headers are correct.

Links

Discussion