Crawl and Scrape icon

Crawl and Scrape

Crawl websites and extract data

Actions3

Extract Links

AI-generated

Overview

This node performs web crawling and scraping to extract all links from a specified URL. It supports crawling with or without a headless browser (Playwright), allowing it to handle JavaScript-heavy sites. The node can use proxy URLs, custom headers, and cookies to customize requests. It is useful for scenarios where you need to gather all hyperlinks from a webpage or multiple pages up to a specified depth, such as for SEO analysis, link auditing, or data collection.

Use Case Examples

  1. Extract all links from a single webpage to analyze its outbound links.
  2. Crawl a website up to a certain depth to collect all internal and external links for site mapping.
  3. Use proxy URLs and custom headers to crawl websites that require specific access configurations.

Properties

Name Meaning
URL The URL to crawl or scrape, required to specify the starting point for link extraction.
Max Depth Maximum depth of crawling, controls how many levels of links to follow during extraction.
Proxy URLs A list of proxy URLs to use for requests, allowing crawling through proxies for anonymity or access control.
Use Browser (Playwright) Whether to use a headless browser for crawling, useful for sites that require JavaScript execution.
Header Input Type How to provide custom headers: as a JSON object or a raw string.
Custom Headers (JSON) Custom headers to send with requests, provided as a JSON object.
Custom Headers (Raw String) Custom headers to send with requests, provided as a raw string supporting key-value format.
Cookies Input Type How to provide cookies: as a JSON object or a raw string.
Cookies (JSON) Cookies to send with requests, provided as a JSON object.
Cookies (Raw String) Cookies to send with requests, provided as a raw string (e.g., key=value; key2=value2).

Output

JSON

  • status - Status of the crawling operation, e.g., 'success'.
  • message - Message describing the result of the operation.
  • data
    • url - The URL that was crawled.
    • links - An array of extracted links (URLs) found on the crawled page(s).

Dependencies

  • crawlee library for crawling and scraping
  • cheerio for HTML parsing when not using browser

Troubleshooting

  • Ensure the URL is valid and accessible; invalid URLs or network issues can cause failures.
  • If using proxies, verify proxy URLs are correct and proxies are reachable.
  • When using the browser option, ensure the environment supports Playwright and necessary dependencies are installed.
  • Custom headers and cookies must be correctly formatted; malformed headers or cookies can cause request failures.
  • Timeouts may occur if the target site is slow or unresponsive; consider adjusting timeout settings if available.

Links

  • Crawlee Documentation - Official documentation for the Crawlee library used for crawling and scraping.
  • Playwright Documentation - Documentation for Playwright, the headless browser automation tool used optionally by the node.

Discussion