Olostep Web ScraperInstall

Summary

Batch scrape multiple URLs simultaneously by submitting a JSON array of URL and custom_id objects to retrieve their web content in the specified format.

Inputs

• URLs to Scrape (required) — A JSON array string where each object must include a 'url' field and a 'custom_id' for identification, e.g., [{"url":"https://example.com","custom_id":"site1"}].
• Output Format — Select the format for the scraped content output; options are HTML, Markdown, JSON, or Plain Text. Defaults to Markdown.
• Country Code — Optional two-letter country code (e.g., US, GB, CA) to enable location-specific scraping.
• Wait Before Scraping (Milliseconds) — Optional wait time in milliseconds before scraping each URL to allow for dynamic content rendering; range 0 to 10,000 ms.
• Parser — Optional parser ID string to specify specialized content extraction (e.g., '@olostep/amazon-product').

Output shape

a single item with JSON containing batch metadata and submitted URLs

Returns metadata including batch ID, status (usually 'processing'), total URLs, timestamps, output format, country, parser, and an array of URLs with their custom IDs. Does not return scraped contents immediately; instead, returns processing status. Maximum batch size is 100,000 URLs. Input JSON must parse correctly or an error is thrown.