Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Extract (Legacy) → Extract structured data from websites using AI

AI-generated

Overview

The Firecrawl node enables users to scrape, crawl, map, search, and extract structured data from websites using the Firecrawl API. Specifically, the 'Extract (Legacy)' resource with the 'Extract structured data from websites using AI' operation allows users to input URLs (supporting glob patterns) and provide natural language prompts and JSON schemas to guide AI-driven data extraction from web pages. This node is beneficial for automating data collection from multiple web pages, such as extracting product details from e-commerce sites or gathering article content from blogs.

Use Case Examples

  1. Extract product names, prices, and descriptions from multiple e-commerce pages using URL patterns and a JSON schema.
  2. Scrape and extract structured data from news articles by specifying a prompt and schema to capture headlines, authors, and publication dates.

Properties

Name Meaning
URLs List of URLs or URL patterns to extract data from, supporting glob patterns for multiple pages.
Prompt Natural language instructions guiding the AI on what data to extract from the web pages.
Schema JSON Schema defining the structure and types of the data to be extracted by the AI.
Ignore Sitemap Option to skip sitemap discovery for faster extraction when exact URL patterns are known.
Include Subdomains Whether to include subdomains matching the URL patterns in the extraction.
Enable Web Search Allow the AI to search the web for additional information not found on the page to enrich extracted data.
Show Sources Include source URLs and page sections where each piece of data was found for verification or debugging.
Scrape Options Settings to control how content is extracted, including output formats, content filtering, timing, caching, browser emulation, and pre-scraping actions.
Additional Fields Custom JSON properties or additional fields to include in the request body.
Use Custom Body Option to use a fully custom request body instead of the standard parameters.
Request Options Node-level HTTP request options such as batching, SSL handling, proxy, and timeout settings.

Output

JSON

  • data - Extracted structured data from the specified URLs according to the prompt and schema.
  • sources - Optional source URLs and page sections where each piece of data was found, if 'Show Sources' is enabled.

Dependencies

  • Firecrawl API key credential

Troubleshooting

  • Ensure the Firecrawl API key credential is correctly configured and valid.
  • Verify that the URLs provided are accessible and correctly formatted, especially when using glob patterns.
  • If extraction results are incomplete or incorrect, refine the natural language prompt and JSON schema to better guide the AI.
  • Check network connectivity and proxy settings if requests fail or time out.
  • Enable 'Show Sources' to debug and verify where data is being extracted from.

Links

Discussion