DataForSEO OnPage APIInstall

Summary

Retrieves a list of pages on the target domain that are blocked from being indexed by search engines via robots.txt, HTTP headers, or meta tags.

Inputs

• target (required) — Domain without https:// and www.
• max_crawl_pages (required) — Maximum number of pages to crawl
• start_url — Absolute URL to start crawl; if set and max_crawl_pages=1 crawls only that page
• force_sitewide_checks — Enable sitewide checks when crawling a single page
• priority_urls — Up to 20 URLs to crawl bypassing the queue
• max_crawl_depth — Maximum crawl depth
• crawl_delay — Delay between hits in ms (default 2000)
• store_raw_html — Store HTML of crawled pages?
• enable_content_parsing — Parse content on crawled pages?
• support_cookies — Support cookies on crawled pages?
• accept_language — Language header for accessing the website
• custom_robots_txt — Custom robots.txt settings
• robots_txt_merge_mode — Merge with or override robots.txt settings (merge|override)
• custom_user_agent — Custom user agent
• browser_preset — Preset for browser screen parameters (empty|desktop|mobile|tablet)
• browser_screen_width — Browser screen width
• browser_screen_height — Browser screen height
• browser_screen_scale_factor — Browser screen scale factor
• respect_sitemap — Respect sitemap when crawling?
• custom_sitemap — Custom sitemap URL
• crawl_sitemap_only — Crawl only pages indicated in the sitemap?
• load_resources — Load resources?
• enable_www_redirect_check — Check if the domain implemented the WWW redirection?
• enable_javascript — Load the scripts available on a page?
• enable_xhr — Enable XMLHttpRequest on a page?
• enable_browser_rendering — Emulate browser rendering to measure Core Web Vitals?
• disable_cookie_popup — Disable the cookie popup?
• custom_js — Custom JavaScript
• validate_micromarkup — Enable microdata validation?
• allow_subdomains — Include pages on subdomains?
• allowed_subdomains — Subdomains to crawl
• disallowed_subdomains — Subdomains not to crawl
• check_spell — Check spelling?
• check_spell_language — Language of the spell check
• check_spell_exceptions — Words excluded from spell check
• calculate_keyword_density — Calculate keyword density for the target domain?
• checks_threshold — Custom threshold values for checks
• disable_sitewide_checks — Prevent certain sitewide checks from running
• disable_page_checks — Prevent certain page checks from running
• switch_pool — Switch proxy pool?
• return_despite_timeout — Return data on pages despite the timeout error?
• filters — Array of results filtering parameters (JSON string)
• limit — Max number of results to return (up to 1000), default 50
• offset — Offset for pagination, default 0

Output shape

DataForSEO OnPage API response for /on_page/non_indexable endpoint; the list of non‑indexable pages is in tasks[0].result.

Each item typically contains fields such as url, reason, etc.

Links

• n8n community nodes installation guide