Crawl4AI Plus AdvancedInstall

Summary

Extract semantically similar content clusters from a specified URL using cosine similarity clustering with configurable sentence embedding models and clustering parameters.

Inputs

• URL (required) — The URL to extract content from using semantic similarity clustering.
• Semantic Filter (required) — Keywords or topic to filter content semantically (e.g., product reviews, user feedback).
• Clustering Options — Parameters to control clustering such as linkage method, max distance threshold, model name for embeddings, similarity threshold, number of top clusters to return, verbosity, and minimum word count.
• Options — Additional output options like whether to normalize whitespace in extracted text and whether to include the full crawled markdown text.
• Browser & Session — Configuration for the browser session to perform crawling, including browser type, user agent, cookies, stealth mode, proxy, headless mode, timeouts, viewport size, and other advanced settings to bypass bot detection or customize crawling behavior.
• Crawl Settings — Settings to customize crawling behavior such as anti-bot modes, resource blocking, cache mode, respecting robots.txt, CSS selectors for extraction, delays, link filtering, JavaScript execution, maximum retries, waiting conditions, and word count thresholds.

Output shape

a list of top-ranked content clusters, each containing semantically similar extracted content blocks from the target URL.

Requires the 'crawl4ai:all' Docker image with torch and sentence-transformers support; the standard latest image lacks these dependencies. The output clusters are filtered by a minimum similarity threshold and limited by the top K clusters parameter. Verbose mode enables detailed clustering logs. Full crawled markdown text can optionally be included in the output.

Links

• n8n community nodes installation guide