Actions13
Deduplicate (Fuzzy)
AI-generatedSummary
Remove duplicate records from input data by performing fuzzy matching on specified fields above a similarity threshold to identify duplicates and keep one representative record from each duplicate group.
Inputs
- Fields to Check (required) — Comma-separated list of field names used to compare and detect duplicates using fuzzy matching (e.g., 'firstName, lastName, email').
- Fuzzy Threshold (required) — A number from 0.0 to 1.0 defining minimum similarity for records to be considered duplicates (default 0.8 means 80% similarity).
- Output Duplicate Info — Boolean to include metadata about removed duplicates and duplication statistics in the output (false by default).
- Options — Optional settings to control additional behavior including: trackChanges (add _changes field showing modifications), continueOnFail (continue processing on errors), skipUnchanged (omit unchanged items), caseInsensitiveFields (match field names ignoring case, default true), and debugMode (include detailed debug info in _debug field).
Output shape
a list of deduplicated records in the same format as input, each a single JSON object; optionally includes a _deduplicationInfo field on the first item with metadata if Output Duplicate Info is enabled.
If Output Duplicate Info is enabled, the first output item includes _deduplicationInfo with counts of original vs deduplicated records, removed count, duplicate groups found, fields checked, and the threshold used. Tracking changes adds a _changes array showing operation summary. The fuzzy threshold must be between 0.0 and 1.0. The node performs case-insensitive matching of specified fields by default for detecting duplicates. Errors thrown if required fields are missing or threshold is out of range.
Examples
Example 1: Remove fuzzy duplicates based on 'name' and 'email' fields with a similarity threshold of 0.85 and include duplicate info in output.
Fields to Check='name,email'; Fuzzy Threshold=0.85; Output Duplicate Info=true; default options.