Website Content Crawler by Apify icon

Website Content Crawler by Apify

Crawl websites and extract text content to feed AI agents, LLM applications, vector databases, or RAG pipelines.

Actions2

Website Content Crawler → Crawl Website

AI-generated

Overview

The Website Content Crawler by Apify is designed to crawl websites starting from specified URLs and extract text content. It is useful for feeding AI agents, large language model (LLM) applications, vector databases, or retrieval-augmented generation (RAG) pipelines. The node supports crawling sub-pages, considering URLs from sitemaps, and choosing different crawling methods including adaptive browser and raw HTTP client.

Use Case Examples

  1. Crawl a blog starting from its main page and extract all article content for analysis.
  2. Use the crawler to gather product descriptions from an e-commerce site for a recommendation engine.
  3. Feed crawled website content into a vector database for semantic search applications.

Properties

Name Meaning
Authentication Choose which authentication method to use for accessing the Apify API (API Key or OAuth2).
Start URLs One or more URLs where the crawler will start crawling. The crawler will also crawl sub-pages of these URLs unless overridden by other options.
Consider URLs From Sitemaps Whether the crawler should look for sitemaps at the domains of the start URLs and enqueue matching URLs for crawling.
Crawler Type Select the crawling method: adaptive switching between browser and raw HTTP (recommended), headless browser with Firefox and Playwright, or raw HTTP client (fastest but no JavaScript rendering).
Max Crawling Depth Maximum number of link levels the crawler will follow starting from the start URLs. Depth 0 means only the start URLs are crawled.
Max Pages Maximum number of pages to crawl in total, including start URLs, pagination pages, and pages with no content.

Output

JSON

  • json - The extracted content and metadata from the crawled web pages.

Dependencies

  • Requires Apify API access via API key or OAuth2 authentication.

Troubleshooting

  • If the crawler does not start, check that the authentication credentials are correctly configured and valid.
  • If pages are missing or not rendered correctly, try switching the crawler type to use the headless browser option to handle JavaScript-heavy sites.
  • If the crawl stops prematurely, verify the maxDepth and maxPages settings to ensure they are not too restrictive.
  • Errors related to network or API limits may require checking the Apify account usage and quotas.

Links

Discussion