Crawlbase icon

Crawlbase

Crawl a URL with Crawlbase Crawling API

Crawlbase

AI-generated

Summary

Crawls a URL through the Crawlbase API using GET, POST, or PUT, and returns the response with status, headers, body, and extracted metadata. Choose a fixed URL or read one from a named field on each input item; optional settings control response format and Crawlbase request behavior.

Inputs

  • URL Source (required) — Choose a fixed URL from the URL field or read a URL from each input item.
  • URL (required) — When using a fixed URL, provide an HTTP or HTTPS URL. The node validates it before sending the request.
  • Input Field Name (required) — When using input items, name the JSON field containing each HTTP or HTTPS URL. Defaults to url.
  • Method (required) — Select GET, POST, or PUT. For POST and PUT, an optional Request Body is sent when nonempty; JSON-shaped valid text is sent as application/json, otherwise as application/x-www-form-urlencoded.
  • Response Format (required) — Choose HTML, JSON, or Markdown. The selected format is passed to Crawlbase; format-specific options include JSON pretty printing and Markdown readability.
  • Options — Optional Crawlbase settings include AJAX and page waits, asynchronous crawling, autoparse, cookies and cookie sessions, country, CSS clicks, custom success codes, device, response cookies and headers, screenshots and dimensions, scraper, scrolling, Cloud Storage, Tor, user agent, and custom request headers. Timeout sets the HTTP client timeout in milliseconds (default 90000), not an API parameter.
  • Request Body (POST/PUT) — Optional body for POST or PUT requests; ignored for GET and when empty.

Output shape

One output item per processed input item, each containing a JSON object with statusCode, headers, body, and metadata.

metadata may contain originalStatus, cbStatus, and url when available. With Continue On Fail enabled, failed requests produce an item with statusCode 0, empty body, headers, and metadata, plus an error message. Async and storage options can return Crawlbase identifiers or URLs in the response body.

Examples

Example 1: Crawl one page with a fixed URL

Select From Parameter Below, enter an HTTP or HTTPS URL, and choose a response format such as HTML.

Example 2: Crawl URLs from incoming items

Select From Input Item Field and set the field name containing each URL (default: url). The node processes each item and keeps its item pairing.

Example 3: Send a POST or PUT request

Select POST or PUT and optionally provide a body. Valid JSON objects or arrays are sent as application/json; other body text is sent as form data.

Links

Discussion