CRW

Scrape, crawl, and extract web data with CRW — the open-source web scraper built for AI agents

Actions6

Scrape

AI-generated

Summary

Scrape a single URL and return its content in multiple formats using the CRW API.

Inputs

  • url (required) — The URL to scrape.
  • formats — Output formats to return (e.g., 'markdown', 'html', 'links').
  • onlyMainContent — Whether to strip navigation, footer, sidebar and return only the main content.
  • renderJs — JavaScript rendering mode: 'auto', 'force', or 'http'.
  • waitFor — Milliseconds to wait after page load before capturing content.
  • cssSelector — CSS selector to extract specific content.
  • xpath — XPath expression to extract specific content.
  • includeTags — Comma-separated CSS selectors of elements to include.
  • excludeTags — Comma-separated CSS selectors of elements to exclude.
  • headers — Custom HTTP headers to send with the request.
  • proxy — Proxy URL to use for this request.
  • stealth — Whether to enable stealth mode for this request.
  • Schema — JSON Schema for structured data extraction (requires 'JSON' output format).

Output shape

A single record containing the scraped content.

The shape depends on the selected output formats (e.g., 'markdown', 'html', 'links', ''). If '' is selected, the content is structured according to the provided JSON Schema.

Examples

Example 1: Scrape a single URL and return the content as Markdown.

URL: https://example.com, Formats: markdown, Only Main Content: true

Example 2: Scrape a URL and return only the raw HTML.

URL: https://example.com, Formats: rawHtml, Only Main Content: false

Example 3: Extract specific content using a CSS selector.

URL: https://example.com, Formats: html, Additional Options: cssSelector #main

Links

Discussion