Firecrawl icon

Firecrawl

Scrape, crawl, map, search, and extract structured data from websites using Firecrawl API

Actions31

Crawling → Preview crawl params from prompt

AI-generated

Overview

This node operation allows users to preview crawl parameters generated from a natural language prompt for a specified webpage URL. It is useful for scenarios where users want to understand or verify how the Firecrawl API interprets their crawl instructions before executing a full crawl. For example, a user can input a URL and a descriptive prompt about the desired crawl behavior, and the node will return a preview of the crawl parameters that would be used.

Use Case Examples

  1. Preview crawl parameters for https://firecrawl.dev with a prompt describing the desired data extraction.
  2. Verify crawl settings generated from a natural language prompt before running a large-scale crawl.

Properties

Name Meaning
URL The target webpage URL to scrape or crawl. Must be a valid HTTP/HTTPS URL and publicly accessible or accessible with configured authentication.
Prompt Natural language prompt describing the desired crawl behavior to generate crawl parameters.
Use Custom Body Boolean flag indicating whether to use a custom request body.
Request Options Collection of advanced request options including batching, SSL certificate validation, proxy settings, and timeout duration.

Output

JSON

  • crawlParamsPreview - Preview of the crawl parameters generated from the prompt and URL input.

Dependencies

  • Firecrawl API key credential

Troubleshooting

  • Invalid URL format or inaccessible URL may cause request failures. Ensure the URL is correct and publicly accessible or properly authenticated.
  • Improperly formatted or unclear prompt may result in unexpected crawl parameters. Refine the natural language prompt for clarity.
  • Network issues or proxy misconfiguration can cause request timeouts or failures. Verify network connectivity and proxy settings.
  • SSL certificate validation errors can occur if the target server uses an invalid or self-signed certificate. Use the 'Ignore SSL Issues' option cautiously to bypass this.

Links

Discussion