MagicDoor Kafka Event Type Trigger icon

MagicDoor Kafka Event Type Trigger

Consume messages from a Kafka topic

MagicDoor Kafka Event Type Trigger

AI-generated

Overview

This node is a Kafka event type trigger designed to consume messages from a specified Kafka topic. It filters messages based on a given event type and supports various consumer group and processing options. It is useful for workflows that need to react to specific Kafka events, such as customer onboarding approvals or other domain-specific events, enabling real-time data processing and automation.

Use Case Examples

  1. Trigger a workflow when a 'CustomerOnboardingApprovedEvent' is published to a Kafka topic.
  2. Consume messages from a Kafka topic starting from the beginning to process historical data.
  3. Filter Kafka messages by event type and process them in parallel or sequentially based on configuration.

Properties

Name Meaning
Topic Name of the Kafka topic to consume messages from.
Group ID Unique identifier for the consumer group, ensuring distinct consumption for each workflow.
Event Type The specific event type to filter messages by, only messages matching this type will trigger the workflow.
Options Additional configuration options for the Kafka consumer, including topic creation allowance, commit thresholds, heartbeat intervals, message reading preferences, processing mode, and output format.

Output

JSON

  • message - The Kafka message content, parsed from the consumed message.
  • headers - Optional Kafka message headers if configured to be returned.
  • topic - The Kafka topic from which the message was consumed.

Dependencies

  • KafkaJS library for Kafka client functionality
  • Kafka credentials including brokers, client ID, SSL, and optional SASL authentication

Troubleshooting

  • Ensure Kafka credentials are correctly configured, including brokers and authentication details.
  • Verify that the consumer group ID is unique to avoid conflicts in message consumption.
  • Check that the event type filter matches the message type field correctly, considering case sensitivity.
  • If messages are not being processed, confirm the topic name and that messages exist in the topic.
  • Adjust session timeout and heartbeat interval settings to maintain a stable consumer session.

Links

Discussion