Unified Scraper

Generic unified scraper - reusable for any scraper service

Actions3

Get by ID

AI-generated

Summary

Fetches a specific piece of content by ID or URL from a configured scraper service via POST to its /get endpoint, with retry/backoff and optional insertion of the results into scraper-db-service.

Inputs

  • scraperBaseUrl (required) — Base URL of the scraper service (e.g. http://scraper-x:8082). The node appends /get.
  • apiKey — API key for the scraper. If set, it is sent as both X-API-Key and Authorization: Bearer headers.
  • id — Content ID or URL, sent as id in the JSON body. Evaluated per input item, so it can be an expression.
  • limit — Maximum number of results (1-100, default 20). Sent as limit in the body.
  • insertToDb — If true, and the credential has a scraperDbApiKey, posts the scraper response to {dbBaseUrl}/api/ingest. Defaults to false.
  • maxRetries — Retry attempts (0-5, default 3) for the scraper call and the DB insert. Uses exponential backoff starting at 1s and capped at 16s.

Output shape

One output item per input item. Each has json fields: success, scraper (base URL), operation, results (raw scraper response), dbInserted, dbStats ({added, updated, skipped} when the insert succeeded), errors (attempt messages, if any) and retryCount.

A failed DB insert does not fail the node. dbInserted is false and the error is added to errors. DB insert only runs if the unifiedScraperApi credential provides scraperDbApiKey (the DB URL defaults to http://scraper-db-service:8088). If the scraper call fails after all retries, the node throws, unless Continue On Fail is enabled. In that case the item has success:false and an error message. The node requires the unifiedScraperApi credential. Parameters are read once from item 0, except id and limit, which are read per item.

Examples

Example 1: Fetch a single post by URL from a scraper service

Operation: Get by ID; Scraper Base URL: http://scraper-x:8082; API Key set; ID / URL: https://example.com/post/123; Insert to Database off.

Example 2: Fetch items by ID from earlier nodes and store them in the DB

ID / URL: {{$json.postId}}; Insert to Database on; Max Retries 3; requires scraperDbApiKey in the credential.

Links

Discussion