Crawl and Scrape icon

Crawl and Scrape

Crawl websites and extract data

Actions3

Extract Text

AI-generated

Overview

This node performs web crawling and scraping to extract text content from a specified URL. It supports using either a headless browser (Playwright) for JavaScript-heavy sites or a lightweight HTTP crawler (Cheerio) for simpler pages. The node can handle custom headers, cookies, and proxy configurations, making it suitable for extracting textual data from various web pages, including those requiring authentication or behind proxies.

Use Case Examples

  1. Extracting article text from news websites for content analysis.
  2. Scraping product descriptions from e-commerce sites for price comparison.
  3. Gathering textual data from web pages that require JavaScript rendering, such as dynamic content or single-page applications.

Properties

Name Meaning
URL The URL to crawl or scrape, required to specify the target web page.
Proxy URLs A list of proxy URLs to route requests through, useful for bypassing IP restrictions or accessing geo-restricted content.
Use Browser (Playwright) Whether to use a headless browser for crawling, necessary for sites that require JavaScript execution.
Header Input Type How to provide custom HTTP headers, either as a JSON object or a raw string.
Custom Headers (JSON) Custom HTTP headers provided as a JSON object to include in requests.
Custom Headers (Raw String) Custom HTTP headers provided as a raw string, supporting 'Key: Value' format or alternating lines.
Cookies Input Type How to provide cookies, either as a JSON object or a raw string.
Cookies (JSON) Cookies provided as a JSON object to include in requests.
Cookies (Raw String) Cookies provided as a raw string, e.g., 'key=value; key2=value2'.

Output

JSON

  • status - Status of the extraction operation, e.g., 'success'.
  • message - Descriptive message about the extraction result.
  • data
    • url - The URL from which text was extracted.
    • text - The extracted text content from the web page body.
    • title - The title of the web page, if available.
    • description - The content of the meta description tag, if available.

Dependencies

  • crawlee library for crawling and scraping
  • cheerio library for HTML parsing
  • playwright for headless browser crawling

Troubleshooting

  • Ensure the URL is valid and accessible; invalid URLs or network issues can cause failures.
  • If using the browser option, ensure the environment supports Playwright and its dependencies.
  • Proxy URLs must be correctly formatted; incorrect proxy settings can block requests.
  • Custom headers and cookies must be properly formatted JSON or raw strings; malformed inputs can cause request errors.
  • Timeouts may occur on slow or heavily loaded sites; consider increasing timeout settings if supported.

Links

Discussion