Unified Scraper

Generic unified scraper - reusable for any scraper service

Actions3

Search

AI-generated

Summary

Sends a search query to a self-hosted scraper service via POST {baseUrl}/search and returns the scraper's response, optionally ingesting the results into a scraper-db-service database.

Inputs

  • Scraper Base URL (required) — Base URL of the scraper service (e.g. http://scraper-x:8082). '/search' is appended to it.
  • API Key — If set, sent to the scraper as both an X-API-Key header and an Authorization: Bearer header.
  • Query — Search keyword or query, sent as query in the JSON body. Evaluated per input item.
  • Limit — Maximum number of results (1-100, default 20). Sent as limit in the request body.
  • Insert to Database — When enabled, POSTs results to {dbBaseUrl}/api/ingest. This only happens if the unifiedScraperApi credential supplies a scraperDbApiKey. The DB URL comes from the credential's scraperDbBaseUrl, defaulting to http://scraper-db-service:8088.
  • Max Retries — Retry attempts (0-5, default 3) for both the scraper call and the DB insert. Backoff is exponential: 1s, 2s, 4s and so on, capped at 16s.

Output shape

One output item per input item. Each has json fields: success, scraper (base URL), operation, results (the raw scraper response), dbInserted, dbStats ({added, updated, skipped}, only when the insert succeeded), errors (list of retry/DB error messages, if any), and retryCount.

Scraper and database settings are read once from the first item, while the query and limit are evaluated per item. A DB insert failure does not fail the node. It is reported in errors with dbInserted false. If the scraper call fails after all retries, the node throws, unless Continue On Fail is on. In that case it emits an item with success false and an error message. The node requires the unifiedScraperApi credential. The postgres credential is optional and is not used by this code.

Examples

Example 1: Search a scraper service for a keyword and return up to 50 results

Scraper Base URL=http://scraper-x:8082, API Key=, Query='ai agents', Limit=50, Insert to Database=false

Example 2: Search and persist results to the scraper DB with retries

Query={{$json.keyword}}, Limit=20, Insert to Database=true, Max Retries=5; unifiedScraperApi credential must include scraperDbApiKey

Links

Discussion