Robots.txt Checker icon

Robots.txt Checker

Analyze robots.txt files for SEO issues, blocked resources, and crawl directives. By John Rippy.

Robots.txt Checker

AI-generated

Summary

Checks a website’s robots.txt directives through the Apify Robots.txt Checker actor, optionally checking referenced sitemaps. For each input item, returns the actor run status and, when it succeeds, the dataset results.

Inputs

  • URL (required) — Website URL to check.
  • User Agent — Bot identity whose directives to check; defaults to Googlebot. Options include Bingbot, all bots (*), GPTBot, Claude-Web, or Custom.
  • Custom User Agent — User agent string used when User Agent is set to Custom.
  • Check Sitemap — Whether to also check referenced sitemaps; defaults to true.
  • Demo Mode — Whether to run with sample data instead of performing the actual check; defaults to false.
  • Timeout (seconds) — How long the node waits for the Apify run to finish; defaults to 120 seconds.

Output shape

One output item per input item, each containing JSON.

A successful run returns runId, status, and results from the Apify dataset. A run that does not succeed returns runId, status, and a message. Requires Apify API credentials.

Links

Discussion