Link CrawlerInstall

Summary

Crawl links starting from a given URL, optionally including external domains and filtering results by a regex pattern.

Inputs

• URL (required) — Starting page URL to begin crawling links from.
• Crawl External Links? — Boolean flag to determine whether to crawl links outside the starting URL's domain.
• Search Pattern — Regex pattern to filter the discovered links; supports special syntax where '(.)' converts to '(.*)'.

Output shape

a list of objects each containing a 'url' field representing a discovered link matching the criteria

The crawl searches up to 3 link levels deep and returns unique URLs as separate JSON items. Errors during execution are returned as error messages if 'continueOnFail' is enabled.