Crawl4aiInstall

Summary

Crawl specified URLs using an advanced browser-based crawler configured with customizable browser settings, anti-detection measures, and authentication methods, and extract complex data via large language model (LLM) techniques.

Inputs

• URL Configuration (required) — One or more URLs to crawl, each specified as a string URL.
• Browser Configuration — Settings for the crawling browser including headless mode, viewport dimensions, and custom user agent string.
• Anti-Detection Settings — Options to enable undetected or stealth browser modes and configure CAPTCHA bypass strategies to prevent detection during crawling.
• Authentication Configuration — Authentication details if needed for the targeted websites, supporting Basic Auth, Form Auth, OAuth2, API Key, or Session Cookie, including credentials and login URLs.
• Advanced Configuration — Advanced options controlling retry attempts, request timeout, concurrency level, proxy usage, and debug logging.

Output shape

A list of JSON objects each representing the extraction result or error for a single URL.

Each output item contains detailed crawl metadata including URL, operation type, browser configuration, anti-detection settings, authentication status, advanced config parameters, success flag, and timestamp. Errors include an error message and success=false. Multiple URLs produce multiple output items, paired by input item index.