Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Crawling → Get crawl errors

AI-generated

Overview

This node interacts with the Firecrawl API to retrieve crawl errors for a specific crawl job. It is useful for monitoring and debugging web crawling tasks by fetching error details associated with a given crawl ID. For example, users can input a crawl job ID to get a list of errors encountered during that crawl, helping to identify issues such as failed page loads or extraction problems.

Use Case Examples

  1. A user wants to check the errors from a recent web crawl to troubleshoot why some pages were not processed correctly.
  2. A developer integrates this node in an automation workflow to periodically fetch and log crawl errors for quality assurance.

Properties

Name Meaning
Crawl ID The unique identifier of the crawl job for which to retrieve errors.
Use Custom Body Flag to determine whether to send a custom request body with the API call.
Request Options Additional settings for the HTTP request such as batching, SSL certificate validation, proxy configuration, and timeout settings.

Output

JSON

  • errors - List of crawl errors returned by the Firecrawl API for the specified crawl job.

Dependencies

  • Firecrawl API
  • An API key credential for Firecrawl API authentication

Troubleshooting

  • Ensure the Crawl ID is correct and corresponds to an existing crawl job; otherwise, the API may return an error or empty results.
  • If using a proxy, verify the proxy URL and credentials are correct to avoid connection failures.
  • If SSL issues occur, consider enabling the option to ignore SSL certificate validation, but be aware of security risks.
  • Timeout errors may occur if the server is slow to respond; increase the timeout setting if necessary.

Links

Discussion