Parquet icon

Parquet

Read and write Apache Parquet files with multi-cloud storage support. Created by Gerd First-Grüttner.

Actions4

Merge

AI-generated

Summary

Merge multiple Apache Parquet files from specified sources with compatible schemas into a single Parquet file saved to a chosen storage location.

Inputs

  • Sources (required) — One or more source entries, each specifying the storage type (Local File, Azure Blob Storage, AWS S3, or Google Cloud Storage) and the path or pattern to match Parquet files to merge. Wildcards are supported but only non-recursively (in filename only).
  • Output Storage Type (required) — The storage destination type for the merged Parquet file (Local File, Azure Blob Storage, AWS S3, or Google Cloud Storage).
  • Output File Path or Location (required) — Depending on the output storage type, provide the absolute file path (local file) or container/bucket and path/key details for where the merged file will be saved.
  • Options — Collection of options including max files to merge (default 1000), schema mismatch handling (collect all mismatches or fail fast), schema merge mode (strict requires identical schemas, union adds missing columns with nulls, intersection keeps only common columns), type conflict handling (error, coerce, or null), keys to include or exclude (supports dot paths), row group size for the output file, and enabling progress logging.

Output shape

a single record

Returns JSON indicating success status, number of files merged, total rows merged, and output storage type. On schema mismatch or other errors, may throw or return failure info depending on 'Continue On Fail' setting.

Links

Discussion