Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Agent → Agent - AI-powered web data extraction (waits for completion)

AI-generated

Overview

This node uses the Firecrawl API to perform AI-powered web data extraction based on a natural language prompt. It can autonomously search the web or focus on specified URLs to extract structured data. The node waits for the extraction process to complete and returns the extracted data. This is useful for scenarios like extracting company founders, pricing information, or product feature comparisons from web pages without manual scraping.

Use Case Examples

  1. Extract the founders of a company and their roles from the web.
  2. Extract pricing information from specific product pages.
  3. Compare features between products by extracting structured data from multiple URLs.

Properties

Name Meaning
Prompt Natural language description of the data to extract, guiding the AI on what information to find.
Specify URLs Boolean to decide whether to specify URLs for the agent to focus on or let it autonomously search the web.
URLs List of URLs to focus the agent on specific pages, supporting multiple formats like comma-separated, newlines, or JSON array.
Schema Type Defines how the output data schema is determined: none, generated from a JSON example, or manually defined.
JSON Example Example JSON object used to generate the output schema automatically.
Schema Manually defined JSON Schema specifying the structure of the extracted data.
Max Wait Time Maximum time in seconds to wait for the agent to complete extraction before returning a job ID for manual status checking.
Use Custom Body Boolean to decide whether to send a custom request body to the agent endpoint.
Custom Body Custom JSON body to send to the agent endpoint, overriding standard properties.
Request Options Additional HTTP request options like batching, SSL handling, proxy, and timeout settings.

Output

JSON

  • data - Extracted structured data from the web based on the prompt and optional schema.
  • jobId - Job ID returned if the max wait time is exceeded, allowing manual status checks.

Dependencies

  • Firecrawl API
  • An API key credential for Firecrawl API authentication

Troubleshooting

  • If the node times out, increase the Max Wait Time or check the job status manually using the returned job ID.
  • Ensure the API key credential is valid and has proper permissions to access the Firecrawl API.
  • If specifying URLs, verify the URLs are correctly formatted and accessible.
  • If using a custom schema, ensure it is valid JSON Schema without unsupported $refs syntax.

Links

Discussion