Scrap
AI-generatedOverview
This node integrates with the Firecrawl Web Crawler API to perform web scraping operations. Specifically, the "Scrap" operation allows users to scrape a single URL for content in various formats such as Markdown or screenshots. It supports advanced options like waiting for JavaScript to load, mobile device emulation, custom HTTP headers, and filtering HTML tags.
Common scenarios where this node is beneficial include:
- Extracting main content from a webpage for further processing or analysis.
- Capturing visual snapshots of webpages via screenshots.
- Automating data collection from websites that require dynamic content loading.
- Filtering scraped content by including or excluding specific HTML tags.
Practical example: A user wants to scrape an article page to extract only the main textual content in Markdown format while excluding navigation bars and footers. They can configure the node to scrape the URL, enable "Only Main Content," and select the Markdown output format.
Properties
| Name | Meaning |
|---|---|
| URL | The web address (URL) of the page to scrape. |
| Formats | Output formats to include. Options: Markdown, Single Screenshot, Full Screenshot. |
| Additional Options | Collection of optional settings: - Only Main Content: Return only main content, excluding header/nav/footer. - Include Tags: Comma-separated list of HTML tags to include. - Exclude Tags: Comma-separated list of HTML tags to exclude. - Custom Headers (JSON): JSON string for custom HTTP headers (e.g., cookies, user-agent). - Wait Time (ms): Milliseconds to wait before scraping to allow JS to load. - Mobile Emulation: Emulate a mobile device during scraping. - Skip TLS Verification: Skip SSL/TLS certificate verification. - Timeout (ms): Request timeout in milliseconds. |
Output
The node outputs a JSON object containing the scraped data. The structure depends on the requested formats but generally includes:
- For Markdown format: The main content converted into Markdown text.
- For Screenshot formats: Image data representing either a single viewport screenshot or a full-page screenshot.
- Other included formats will be present if selected (not applicable here since only "scrape" operation is considered).
If multiple formats are requested, the output JSON will contain corresponding fields for each format.
Binary data (screenshots) is represented appropriately within the output, typically as base64-encoded images or references suitable for downstream binary handling in n8n.
Dependencies
- Requires an API key credential for the Firecrawl Web Crawler API.
- The node makes HTTP POST requests to Firecrawl endpoints (
https://api.firecrawl.dev/v1/scrape). - No additional environment variables are required beyond the API key credential configuration.
Troubleshooting
- Invalid JSON in Custom Headers: If the "Custom Headers (JSON)" property contains invalid JSON, the node throws an error stating "Custom Headers must be valid JSON." To fix, ensure the JSON string is correctly formatted.
- API Errors: If the Firecrawl API returns an error, the node throws an error with the message starting with "Firecrawl Scrape error:" followed by the API's error message. Check the URL validity, API key permissions, and network connectivity.
- Timeouts: If the request times out, consider increasing the "Timeout (ms)" value in Additional Options.
- Empty or Unexpected Output: Verify that the URL is accessible and that the selected formats and options are appropriate for the target page.
Links and References
- Firecrawl Web Crawler API Documentation: https://firecrawl.dev/docs
- n8n Documentation on Creating Nodes: https://docs.n8n.io/integrations/creating-nodes/