Unified Scraper

Generic unified scraper - reusable for any scraper service

Actions3

Trending

AI-generated

Summary

Fetches trending/hot content from a configured scraper service by sending a GET request to its /trending endpoint, optionally inserting the results into a scraper-db-service database.

Inputs

  • Scraper Base URL (required) — Base URL of the scraper service (default http://scraper-x:8082). '/trending' is appended to it.
  • API Key — Key for the scraper service. If set, it is sent as both X-API-Key and Authorization: Bearer headers.
  • Limit — Maximum results (1-100, default 20). Sent as the 'limit' query string parameter.
  • Insert to Database — If true, POSTs the results to {db base URL}/api/ingest. Only happens when the credential supplies a DB API key (scraperDbApiKey); the DB URL defaults to http://scraper-db-service:8088 if the credential omits it.
  • Max Retries — Retry attempts (0-5, default 3) for both the scraper call and the DB insert, with exponential backoff (1s, 2s, 4s, up to 16s).

Output shape

One output item per input item, with json: success, scraper (base URL), operation, results (raw response from the scraper), dbInserted (boolean), dbStats ({added, updated, skipped}, only when the insert succeeded), errors (list of failed attempt messages, if any) and retryCount.

A failed DB insert doesn't fail the item: dbInserted is false and the error is added to errors. If the scraper call fails after all retries, the node throws, or with Continue On Fail returns an item with success:false and an error message. The request is made once per input item with the same settings, and the node requires the 'unifiedScraperApi' credential. The DB insert is skipped silently if Insert to Database is on but no DB API key is configured.

Examples

Example 1: Pull the top 10 trending items from an X scraper each hour.

Scraper Base URL=http://scraper-x:8082, API Key set, Limit=10, Insert to Database=false, Max Retries=3, run from a Schedule Trigger.

Example 2: Collect trending content and store it automatically.

Limit=50, Insert to Database=true, with the unifiedScraperApi credential containing scraperDbApiKey; check dbStats in the output.

Links

Discussion