Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Crawling → Get crawl status

AI-generated

Overview

This node interacts with the Firecrawl API to retrieve the status of a specific crawl job. It is useful for monitoring the progress or completion of web crawling tasks initiated through Firecrawl. For example, users can input a crawl job ID to get real-time status updates on that crawl.

Use Case Examples

  1. A user wants to check if a web scraping job has finished or is still running by providing the crawl ID to this node.
  2. A workflow that triggers alerts or subsequent actions based on the crawl status retrieved from Firecrawl.

Properties

Name Meaning
Crawl ID The unique identifier of the crawl job to get status for, used to specify which crawl's status to retrieve.
Use Custom Body A boolean flag indicating whether to use a custom request body in the API call.
Request Options Additional settings for the HTTP request such as batching, SSL certificate validation, proxy configuration, and timeout duration.

Output

JSON

  • status - The current status of the crawl job retrieved from Firecrawl.
  • crawlId - The ID of the crawl job for which the status was requested.
  • details - Additional details or metadata about the crawl status, if provided by the API.

Dependencies

  • Firecrawl API access with an API key credential

Troubleshooting

  • Ensure the provided Crawl ID is correct and corresponds to an existing crawl job; otherwise, the API may return an error or no data.
  • If using a proxy, verify the proxy URL and credentials are correct to avoid connection failures.
  • Timeout errors may occur if the server is slow to respond; consider increasing the timeout setting in request options.
  • If SSL certificate validation fails and the 'Ignore SSL Issues' option is not enabled, the request may be blocked.

Discussion