Sitemap Crawler
AI-generatedSummary
Fetch the sitemap URLs from the specified domain's robots.txt file or fallback locations, send each sitemap URL to an external Crawly service for analysis, and return a summary of URL counts and modification dates found in the sitemaps.
Inputs
- Domain (required) — The base domain to crawl, which is normalized to a valid URL format. The node will attempt to fetch the robots.txt from this domain to locate sitemap URLs.
Output shape
a list containing one object per input item with JSON properties: 'website' (the normalized domain URL), 'summary' (an array summarizing each discovered sitemap's total URLs and last modified dates), 'error' (a string of concatenated error messages if no sitemaps were found or fetching failed), and 'debug' (detailed raw crawler results keyed by sitemap URL).
The node automatically discovers sitemaps from the domain's robots.txt 'Sitemap:' directives or defaults to common sitemap filenames if none found. It uses a POST request to an internal Crawly service for sitemap analysis. The output summary is empty and 'error' populated if all sitemap checks fail. The node processes all input items independently and returns an array of results in order.
Examples
Example 1: Crawl example.com for sitemaps
Set Domain to 'example.com'. The node fetches https://example.com/robots.txt, extracts sitemap URLs, calls the crawly service for each sitemap, and outputs their URL counts and last modification dates.