AWS Firehose icon

AWS Firehose

Consume AWS Firehose API

putRecordBatch

AI-generated

Overview

This node integrates with the AWS Firehose service to send batches of records or files to a specified delivery stream. It is useful for scenarios where you need to efficiently stream large volumes of data to AWS Firehose for further processing or storage, such as log aggregation, real-time analytics, or data lake ingestion. For example, you can batch multiple JSON records or binary files and send them in one request to optimize throughput and reduce API calls.

Use Case Examples

  1. Sending multiple JSON records as a batch to a Firehose delivery stream for real-time processing.
  2. Uploading multiple binary files as base64 encoded data to Firehose for storage in S3 or other destinations.

Properties

Name Meaning
Delivery Stream Name The name of the AWS Firehose delivery stream to which data will be sent.
JSON Parameters For Files Determines if the files should be passed as a JSON array instead of individual binary properties.
Files Collection of files to send to Firehose, each specified by the binary property name containing the file data.
Files JSON Files to send to Firehose in JSON format, structured as an array of file property names.
JSON Parameters For Records Determines if the records should be passed as a JSON array instead of individual record entries.
Records Collection of data records to send to Firehose, each containing raw data as a string.
Records JSON Records to send to Firehose in JSON format, structured as an array of data entries.

Output

JSON

  • DeliveryStreamName - The name of the delivery stream used in the request.
  • Records - Array of records sent to Firehose, each base64 encoded.
  • Response - The response from the AWS Firehose API after sending the batch.

Dependencies

  • AWS credentials for authentication to access the Firehose API.

Troubleshooting

  • Ensure that the delivery stream name is correct and exists in your AWS account.
  • If using JSON parameters, verify that the JSON is properly formatted and represents an array.
  • Check that the binary property names specified for files exist in the input data; otherwise, an error will be thrown.
  • Errors related to AWS authentication usually indicate missing or incorrect AWS credentials.
  • If the Firehose API returns errors, check the AWS service limits and permissions for the delivery stream.

Links

Discussion