Crawl and Scrape icon

Crawl and Scrape

Crawl websites and extract data

Actions3

Extract HTML

AI-generated

Overview

This node performs web crawling and scraping to extract raw HTML content from a specified URL. It supports using either a headless browser (Playwright) for JavaScript-heavy sites or a lightweight HTTP crawler (Cheerio) for simpler pages. The node can handle custom headers, cookies, and proxy configurations, making it suitable for scraping content from various web environments. Practical use cases include gathering HTML for further parsing, archiving web pages, or extracting metadata like page titles and descriptions.

Use Case Examples

  1. Extract raw HTML content from a news article URL to analyze its structure.
  2. Scrape HTML from a product page on an e-commerce site that requires JavaScript rendering.
  3. Retrieve HTML content behind a proxy or with custom headers for authenticated access.

Properties

Name Meaning
URL The URL to crawl or scrape, required for the operation.
Proxy URLs A list of proxy URLs to route requests through, useful for bypassing restrictions or anonymity.
Use Browser (Playwright) Whether to use a headless browser for crawling, necessary for sites requiring JavaScript execution.
Header Input Type Specifies how custom headers are provided, either as a JSON object or a raw string.
Custom Headers (JSON) Custom headers to send with requests, provided as a JSON object.
Custom Headers (Raw String) Custom headers to send with requests, provided as a raw string in 'Key: Value' format.
Cookies Input Type Specifies how cookies are provided, either as a JSON object or a raw string.
Cookies (JSON) Cookies to send with requests, provided as a JSON object.
Cookies (Raw String) Cookies to send with requests, provided as a raw string (e.g., 'key=value; key2=value2').

Output

JSON

  • status - Status of the extraction operation, e.g., 'success'.
  • message - Informational message about the extraction process.
  • data
    • url - The URL from which HTML was extracted.
    • html - The raw HTML content extracted from the page.
    • title - The title of the page, extracted from the HTML.
    • description - The meta description content of the page, if available.

Dependencies

  • crawlee library for crawling and scraping
  • cheerio library for HTML parsing
  • playwright for headless browser crawling (optional)

Troubleshooting

  • Ensure the URL is valid and accessible; invalid URLs or network issues will cause failures.
  • If using the browser option, ensure the environment supports Playwright and its dependencies.
  • Custom headers and cookies must be correctly formatted; malformed inputs can cause request failures.
  • Proxy URLs must be valid and reachable; incorrect proxy settings can block requests.
  • Common errors include timeouts, network errors, or parsing failures; check logs for detailed messages and adjust settings accordingly.

Links

Discussion