Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Crawling → Crawl a website and scrape all pages

AI-generated

Overview

This node performs a web crawling and scraping operation using the Firecrawl API. It is designed to crawl a specified website URL, follow links according to user-defined rules, and extract content from all pages up to a specified limit. It supports detailed configuration such as delays between requests, concurrency limits, inclusion/exclusion of URL paths, and various scraping options including output formats, content filtering, and browser emulation. This node is useful for scenarios like gathering product information, monitoring website changes, or extracting structured data from multiple pages of a website.

Use Case Examples

  1. Crawl an e-commerce site to extract product details and pricing information.
  2. Scrape a blog to collect all articles and summaries for content analysis.
  3. Monitor a news website for updates by crawling all pages and extracting headlines and article content.

Properties

Name Meaning
URL The target webpage URL to scrape or crawl. Must be a valid HTTP/HTTPS URL and publicly accessible or behind configured authentication.
Prompt Natural language instructions to guide the crawl, specifying content focus, page priority, or extraction goals.
Limit Maximum number of results (pages) to return from the crawl.
Delay Wait time in milliseconds between scraping each page to avoid rate limiting or be polite to servers.
Max Concurrency Maximum number of pages to scrape simultaneously, balancing speed and rate limit risk.
Exclude Paths URL path patterns to skip during crawling, using glob patterns to avoid irrelevant or large sections.
Include Paths URL path patterns to include in the crawl, restricting scraping to specific sections.
Crawl Options Settings controlling sitemap usage, query parameter handling, external link following, and subdomain inclusion.
Scrape Options Configuration for content extraction including output formats (markdown, HTML, JSON, screenshot), content filtering, caching, browser emulation, and pre-scraping actions.
Additional Fields Additional custom JSON properties to include in the request body.
Use Custom Body Option to use a fully custom JSON body for the request instead of the standard parameters.

Output

JSON

  • url - The URL of the crawled page.
  • content - Extracted content from the page in the specified formats.
  • metadata - Metadata about the crawl or page such as timestamps, status, or crawl details.

Dependencies

  • Firecrawl API
  • An API key credential for Firecrawl API authentication

Troubleshooting

  • Ensure the URL is valid and publicly accessible or properly authenticated via headers.
  • Be cautious with high concurrency settings to avoid triggering rate limits on target sites.
  • If the crawl returns incomplete data, check include/exclude path patterns and crawl options like sitemap usage.
  • For JavaScript-heavy sites, increase the wait time after page load or use pre-scraping actions to ensure content is fully rendered.
  • If SSL errors occur, consider enabling the option to skip TLS verification, but be aware of security risks.
  • Use caching options appropriately to balance between fresh data needs and performance.

Links

Discussion