Kafka Batch ConsumerInstall

Summary

Consume batches of messages from a specified Kafka topic within a consumer group, optionally starting from the beginning of the topic, and emit them as workflow items.

Inputs

• Group ID (required) — Consumer group identifier to use for the Kafka consumer.
• Topic (required) — Name of the Kafka topic to consume messages from.
• Batch Size (required) — Number of messages to collect before returning a batch.
• From Beginning — Whether to start consuming messages from the beginning of the topic instead of the latest offset.
• Session Timeout — Timeout in milliseconds for the consumer group session; minimum is 6000 ms.
• Options.Read Timeout — Maximum time in milliseconds to wait when collecting messages before returning the batch, even if Batch Size is not reached.
• Options.Parse JSON — Whether to parse each message value as JSON if possible (default true).

Output shape

a single item containing an array of consumed messages as JSON objects representing each Kafka message in the batch

The operation returns after collecting either the configured batch size of messages or when the read timeout expires, whichever comes first. Each message includes topic, partition, offset, key, value (optionally parsed JSON), timestamp, and headers. The consumer disconnects cleanly after consumption. Errors during consumption raise node execution errors.