Crawl4AI: Basic Crawler icon

Crawl4AI: Basic Crawler

Crawl websites using Crawl4AI

Actions3

Crawl Single URL

AI-generated

Overview

The node "Crawl4AI: Basic Crawler" is designed to crawl websites by fetching and processing the content of a single URL. It leverages a headless browser environment with customizable options such as JavaScript execution, viewport size, user agent, and caching strategies. This node is useful for scenarios where you need to extract web page content, analyze specific parts of a page using CSS selectors, or automate interactions on a page via custom JavaScript code.

Practical examples include:

  • Extracting article content from news websites.
  • Scraping product details from e-commerce pages.
  • Automating data collection from dynamic web pages that require JavaScript rendering.
  • Respecting robots.txt rules to comply with website crawling policies.

Properties

Name Meaning
URL The web address to crawl.
Browser Options Settings related to the headless browser environment:
- Enable JavaScript (true/false)
- Headless Mode (true/false)
- Timeout in milliseconds for loading the page
- User Agent string
- Viewport Width and Height in pixels
Crawler Options Controls how the crawler behaves:
- Cache Mode: Enabled (read/write), Bypass (force fresh fetch), Only (read only)
- Check Robots.txt (true/false)
- CSS Selector to focus on specific content
- Exclude External Links (true/false)
- Excluded Tags (comma-separated HTML tags)
- JavaScript Code to execute after page load
- JavaScript Only Mode (true/false)
- Max Retries for failed requests
- Page Timeout in ms
- Request Timeout in ms
- Session ID to maintain browser state across crawls
- Word Count Threshold to filter content based on length
Options Additional output options:
- Include Media Data (images, videos) (true/false)
- Verbose Response including detailed data like HTML and status codes (true/false)

Output

The node outputs JSON data representing the crawled content of the specified URL. Depending on the options selected, the output may include:

  • Extracted text content filtered by CSS selectors or word count thresholds.
  • Metadata about the page such as HTTP status codes.
  • Optionally included media data like images and videos if enabled.
  • Detailed HTML content and other verbose information if requested.

If media data is included, it will be provided in binary form associated with the relevant JSON entries.

Dependencies

  • Requires an API key credential for the Crawl4AI service to authenticate requests.
  • Uses a headless browser environment internally to render and interact with web pages.
  • No additional external dependencies are explicitly required beyond the configured API authentication.

Troubleshooting

  • Timeouts: If the page takes too long to load, increase the timeout settings under Browser Options or Crawler Options.
  • Empty or incomplete content: Ensure the CSS selector is correct and that JavaScript execution is enabled if the page relies on client-side rendering.
  • Cache issues: If stale data is returned, adjust the cache mode to bypass or disable caching.
  • Robots.txt blocking: If no content is returned and robots.txt checking is enabled, verify that crawling is allowed on the target site.
  • Session management: Use the Session ID option to maintain state across multiple crawls when dealing with multi-step navigation or login-required pages.

Common error messages typically relate to network failures, invalid URLs, or authentication errors with the API key. Verify URL correctness and API credentials configuration in n8n.

Links and References

Discussion