News Crawler params icon

News Crawler params

Build and validate payload for POST /batches

News Crawler params

AI-generated

Summary

Builds and validates a payload for the POST /batches endpoint by defining one or more news crawling tasks (RSS or HTML, predefined or custom) and optional webhook notification settings.

Inputs

  • input — Any incoming data; the node processes each input item and outputs an item with the same index containing the generated payload.

Output shape

Each item contains a JSON object with a 'tasks' array (the configured crawling tasks) and an optional 'webhook' object if a webhook URL is set.

The node does not perform crawling itself; it prepares the payload to be sent to an external microservice.

Examples

Example 1: Crawl a custom RSS feed with Cheerio mode and notify a webhook on completion

Add a Task: Type=RSS custom, Mode=Cheerio, Source Label='MyFeed', URL='https://example.com/feed.xml', set Tags='news,tech'. Under Webhook: URL='https://myhook.example.com/notify'.

Discussion