Actions4
Schema Scan
AI-generatedSummary
Scan multiple Apache Parquet files from specified cloud/local storage sources, analyze their schemas, and return either a merged schema (according to a chosen merge mode) or per-file schema details along with the list of source files.
Inputs
- Sources (required) — Define one or more sources specifying storage type (Local File, Azure Blob Storage, AWS S3, Google Cloud Storage) and file path or wildcard pattern to identify Parquet files to scan.
- Options — Set options to control schema merging such as max number of files, mismatch handling (collect or fail fast), schema merge mode (strict, union, intersection), type conflict handling (error, coerce, null), keys to include or exclude, and progress logging.
- Schema Scan Options — Toggle to include per-file schema details and/or the resolved list of source files in the output.
Output shape
a single record (JSON object) summarizing scan results including merged schema, mismatches, number of files scanned, and optionally per-file schemas and source file list.
If strict schema merge mode is selected and schema mismatches exist, the operation will error or return failure information depending on 'continue on fail'. The sources support wildcards but not recursive patterns. The output structure includes 'success', 'filesScanned', 'schemaMode', 'mergedSchema', and may include 'mismatches', 'sourceFiles', and 'perFileSchemas'.
Examples
Example 1: Scan schemas of Parquet files in an AWS S3 bucket with a wildcard pattern, merging schemas via union mode and reporting mismatches but continuing on fail.
Set multiple sources with storage type AWS S3 and specify bucket and file key with wildcards. Enable include per-file schemas option to see schema of each file. Choose union schema merge mode and collect mismatch handling. The output JSON will contain merged schema and per-file schemas.