Crawl Multiple URLs
AI-generatedOverview
The node "Crawl4AI: Basic Crawler" enables crawling multiple URLs using the Crawl4AI service. It is designed to fetch and extract content from web pages, supporting advanced browser options such as JavaScript execution, headless mode, viewport size, and user agent customization. This node is useful for scenarios like web scraping, content aggregation, SEO analysis, or monitoring website changes.
For example, you can provide a list of URLs to crawl, specify CSS selectors to focus on particular page elements, exclude certain HTML tags, and control caching behavior. The node supports concurrent crawling with configurable limits and can optionally include media data or detailed response information.
Properties
| Name | Meaning |
|---|---|
| URLs | Comma-separated list of URLs to crawl. Example: https://example.com, https://example.org |
| Browser Options | Collection of settings controlling the browser environment during crawling: • Enable JavaScript (true/false) • Headless Mode (true/false) • Timeout in milliseconds • User Agent string • Viewport Width and Height (pixels) |
| Crawler Options | Collection of crawling-specific settings: • Cache Mode: Enabled (read/write), Bypass (force fresh), Only (read only) • Check robots.txt rules (true/false) • CSS Selector to target specific content • Exclude external links (true/false) • Excluded HTML tags (comma-separated) • Max retries for failed requests • Page load timeout (ms) • Network request timeout (ms) • Stream results as they become available (true/false) • Word count threshold to filter content |
| Options | Additional options: • Include media data (images, videos) in output (true/false) • Verbose response including detailed data like HTML and status codes (true/false) • Maximum number of concurrent crawls |
Output
The node outputs JSON data representing the crawled content for each URL. Depending on options, the output may include:
- Extracted text content filtered by CSS selectors and word count thresholds.
- Metadata such as HTTP status codes and response headers if verbose mode is enabled.
- Media data (images, videos) if included.
- Information about excluded or external links based on configuration.
If streaming is enabled, partial results may be emitted progressively as pages are crawled.
No binary data output is indicated by the source code or properties.
Dependencies
- Requires an API key credential for the Crawl4AI service to authenticate requests.
- Relies on the Crawl4AI backend to perform actual crawling and content extraction.
- No other external dependencies or environment variables are explicitly required.
Troubleshooting
Common issues:
- Invalid or missing API key credential will cause authentication failures.
- Providing malformed URLs or unreachable sites may result in request errors or timeouts.
- Setting very low timeouts might cause premature termination of page loads.
- Overly restrictive CSS selectors or high word count thresholds could lead to empty results.
- Exceeding max concurrent crawls may overload the service or cause rate limiting.
Error messages:
- Authentication errors indicate problems with the API key; verify and reconfigure credentials.
- Timeout errors suggest increasing the timeout values in browser or crawler options.
- Network errors may require checking connectivity or URL validity.
- Parsing errors might occur if the CSS selector syntax is invalid.
Links and References
- Crawl4AI Official Website — For API documentation and service details.
- n8n Documentation — General guidance on creating and using custom nodes.
- CSS Selectors Reference — To craft valid selectors for content targeting.