Website Content Crawler by Apify icon

Website Content Crawler by Apify

Crawl websites and extract text content to feed AI agents, LLM applications, vector databases, or RAG pipelines.

Actions2

Website Content Crawler → Scrape URL

AI-generated

Overview

The Website Content Crawler by Apify is designed to crawl websites starting from a specified URL and extract text content. It supports different crawling methods including adaptive switching between browser and raw HTTP, headless browser with Firefox and Playwright, and raw HTTP client. This node is useful for feeding AI agents, large language model (LLM) applications, vector databases, or retrieval-augmented generation (RAG) pipelines with web content.

Use Case Examples

  1. Scraping product descriptions from an e-commerce site to build a product database.
  2. Extracting articles from a news website for sentiment analysis.
  3. Gathering content from documentation sites to create a searchable knowledge base.

Properties

Name Meaning
Authentication Choose which authentication method to use for accessing the Apify API, either API Key or OAuth2.
Start URL The URL of the page where the crawler will start crawling. The crawler will also crawl sub-pages of this URL by default.
Crawler Type Select the type of crawler to use: adaptive switching between browser and raw HTTP (recommended), headless browser with Firefox and Playwright, or raw HTTP client (fastest but no JavaScript rendering).

Output

JSON

  • textContent - Extracted text content from the crawled web pages.
  • url - The URL of the crawled page.
  • error - Error message if the crawling or scraping failed for a particular input item.

Dependencies

  • Requires Apify API access via API key or OAuth2 authentication.

Troubleshooting

  • If the node fails to crawl a page, check the authentication credentials and ensure the API key or OAuth2 token is valid.
  • If JavaScript content is not rendered correctly, try switching the crawler type to 'Headless Browser (Firefox+Playwright)' for better JavaScript support.
  • If the node throws errors related to blocked requests or CAPTCHAs, use the headless browser crawler type which retries blocked requests automatically.

Links

Discussion