n8n-nodes-scraperapi

Community n8n node package for the Geonode Scraper API. Wraps the public REST endpoints (/v1/extract, /v1/map, /v1/crawl, /v1/webhooks, /v1/statistics) and ships a webhook trigger that registers itself against /v1/webhooks automatically.

New credentials default to the production API:

https://scraper.geonode.io

You can point it at another deployment by editing the Base URL field on the credential. Use a staging or preprod URL only for intentional testing, with an API key for that environment. Existing saved credentials keep their configured Base URL; update it manually if you want to switch environments.

See the official API reference for the production base URL and authentication details.

Package structure

n8n-nodes-scraperapi/
├─ credentials/
│  └─ ScraperApiApi.credentials.ts   # X-Api-Key credential, tested against /v1/statistics
├─ nodes/
│  ├─ ScraperApi/
│  │  ├─ ScraperApi.node.ts          # Action node (7 resources, ~20 operations)
│  │  ├─ GenericFunctions.ts         # Auth + retry/backoff helpers + polling helper
│  │  ├─ scraperapi.svg              # Node icon
│  │  └─ descriptions/
│  │     ├─ ExtractDescription.ts    # Form schema for extract: run / getJob / listJobs
│  │     ├─ SearchDescription.ts     # Form schema for search: run / getJob / listJobs
│  │     ├─ BatchDescription.ts      # Form schema for batch: start / getStatus / listJobs / cancel
│  │     ├─ MapDescription.ts        # Form schema for map: discover URLs
│  │     ├─ CrawlDescription.ts      # Form schema for crawl: start / getStatus
│  │     ├─ WebhookDescription.ts    # Form schema for webhook CRUD + deliveries
│  │     └─ StatisticsDescription.ts # Form schema for statistics: get
│  └─ ScraperApiTrigger/
│     ├─ ScraperApiTrigger.node.ts   # Trigger node — auto-registers on activate
│     └─ scraperapi.svg
├─ docker-compose.n8n.yml            # Local n8n that mounts this package as a custom extension
├─ package.json                      # n8n entry points -> dist/**
├─ tsconfig.json                     # Compiles credentials/** + nodes/** -> dist/
├─ gulpfile.js                       # Copies SVG icons into dist/
├─ .eslintrc.js                      # eslint-plugin-n8n-nodes-base rules
└─ index.js                          # Empty — n8n loads via the "n8n" key in package.json

Component descriptions

Credential — ScraperApiApi

X-Api-Key: <key> — your personal API key, sent on every request.

Credential fields:

Clicking Test issues GET {baseUrl}/v1/statistics with the credentials applied. This endpoint is auth-gated.

Action node — ScraperApi

One node with seven resources, each exposing the relevant operations:

When a sync extract returns inline, the response is the raw {data, ...} body. For async (processing_mode=async) the API returns {job_id, status, status_url, estimated_tokens}; if Wait for Completion is on, the node polls status_url until the job hits completed | failed | cancelled and returns that final body.

The shared HTTP helper (GenericFunctions.scraperApiRequest) handles:

• X-Api-Key header injection via the credential
• Up to 5 retries on 429 and 5xx with exponential backoff
• Retry-After header support (seconds or HTTP-date)
• Conversion of non-retryable errors into NodeApiError so n8n surfaces them in the UI

Trigger node — ScraperApiTrigger

Use this when you want a workflow to run whenever the Scraper API emits an event (extract_completed, crawl_completed).

On workflow activation it:

1. Calls POST /v1/webhooks with n8n's public webhook URL and the chosen event type.
2. Stashes the returned id and secret in workflow static data.

On webhook delivery it (optionally) verifies X-Webhook-Signature: sha256=<hex> using HMAC-SHA256 + the stored secret, then emits {headers, body} as workflow input.

On deactivation it calls DELETE /v1/webhooks/{id} to clean up.

Running the node

A ready-made compose file lives at docker-compose.n8n.yml in this directory. It mounts this package into n8n's custom-extensions dir and exposes the editor on http://localhost:5678.

From this directory:

# 1. Install deps and build the package
npm install
npm run build        # produces ./dist/ which docker-compose mounts into n8n

# 2. Start n8n
docker compose -f docker-compose.n8n.yml up -d

The compose file mounts the package directory read-only into /home/node/.n8n/custom/n8n-nodes-scraperapi. n8n reads the n8n key from package.json and loads the compiled JS from dist/**, so npm run build must succeed before starting the container (or before restarting after code changes).

After the container is up:

1. Open http://localhost:5678 and finish the n8n owner setup.
2. Credentials → New → Scraper API.
   • Base URL: https://scraper.geonode.io (the default).
   • API Key: your Geonode Scraper API key for production.
   • Click Test — sends an authenticated GET /v1/statistics request.
3. New workflow → search Scraper API → drop it in.

To pick up code changes:

npm run build
docker compose -f docker-compose.n8n.yml restart n8n

To stop / clean up:

docker compose -f docker-compose.n8n.yml down            # stop, keep data
docker compose -f docker-compose.n8n.yml down -v         # also drop the n8n_data volume

Smoke test

In a workflow, add Scraper API → resource extract → operation Run Extraction:

• URL: https://example.com
• Output Formats: html
• Render JavaScript: off
• Processing Mode: Sync (Block Until Done)

Click Execute Node — you should see a JSON output with data.html populated.

Then try the trigger: in a separate workflow add Scraper API Trigger, pick extract_completed, save and activate. The node will register a webhook against /v1/webhooks. Run an extract against the same deployment with processing_mode=async; when it finishes, the trigger workflow fires.

Scripts reference

npm run build      # tsc + copy SVG icons into dist/
npm run dev        # tsc --watch
npm run lint       # eslint with eslint-plugin-n8n-nodes-base
npm run lintfix    # eslint --fix
npm run format     # prettier on nodes/ and credentials/

Notes specific to this API

• Auth — X-Api-Key header (per-user). See app/api/security/auth_token_handler.py. Billing user resolution happens in app/api/security/user_resolver.py.
• Async extract — processing_mode=async returns 202 with {job_id, status, status_url, estimated_tokens} (see app/api/v1/extract/router.py). The node polls status_url when Wait for Completion is on.
• Throttling (HTTP 429) — scraperApiRequest honors Retry-After and retries up to 5 times with exponential backoff (also covers transient 5xx).
• Errors — surfaced as NodeApiError. The API returns {code, message, correlation_id, retryable} (see app/api/v1/extract/schemas.py).
• Webhook events — extract_completed, crawl_completed (see app/core/webhooks/event_types.py).