Olostep Web ScraperInstall

Summary

Start a web crawl from a specified URL, retrieving content from multiple pages up to a maximum limit, optionally following links, and formatting the results.

Inputs

• Start URL (required) — The starting URL for the crawl. Must include http:// or https://.
• Maximum Pages — Maximum number of pages to crawl (minimum 1, default 10).
• Follow Links — Boolean to specify whether to follow links found on pages (default true).
• Output Format — Format for the scraped content. Options are HTML, Markdown, JSON, or Plain Text (default Markdown).
• Country Code — Optional country code to localize the crawl (e.g., US, GB, CA).
• Parser — Optional parser ID for specialized content extraction on crawled pages.

Output shape

A single item containing metadata about the crawl including its ID, status, start URL, maximum pages, follow links flag, content format, country code (if given), parser (if given), and timestamps.

The output reflects the crawl initiation response from Olostep's API and includes the crawl status (such as 'in_progress'). The actual scraped content is not directly returned by this operation but can be accessed or monitored separately using the crawl ID.

Examples

Example 1: Create a crawl starting at https://example\.com that follows up to 20 pages, follows links, returns content in JSON format, localized for US, using a specialized parser.

Start URL: https://example.com; Maximum Pages: 20; Follow Links: true; Output Format: JSON; Country Code: US; Parser: @olostep/custom-parser