OlyptikInstall

Summary

Start a new web crawl from a specified URL using the Olyptik web crawling API, collecting up to a maximum number of results with optional crawl configuration.

Inputs

• Start URL (required) — The URL to start crawling from.
• Max Results — Maximum number of crawl results to collect, from 1 to 10,000, defaulting to 10.
• Additional Fields — Optional crawl configuration including maximum crawl depth (1-100), whether to use sitemap.xml, whether to crawl the entire website, content deduplication, markdown formatting options (exclude non-main tags, include links), selection of crawl engine type (auto, cheerio for static sites, playwright for dynamic sites), usage of static IP addresses, and crawl timeout in minutes.

Output shape

a single object representing the created crawl, containing details returned from the Olyptik API.

Returns the crawl object as JSON after successfully initiating the crawl. Errors during execution will return error messages unless 'Continue on Fail' is enabled, in which case error details appear in the output.

Examples

Example 1: Start a crawl from 'https://example\.com' with up to 20 results and default settings.

Set Start URL to https://example.com and Max Results to 20; leave Additional Fields as default.

Example 2: Initiate a crawl of the entire website at 'https://mysite\.com' using the Playwright engine and a timeout of 30 minutes.

Set Start URL to https://mysite.com, enable 'Entire Website', select 'Playwright' for Engine Type, and set Timeout to 30.