FlowEngine Data Standardize & CleanInstall

Summary

Remove fuzzy duplicates from the input items by comparing specified fields and grouping records with similarity above the given threshold.

Inputs

• Fields to Check (required) — Comma-separated list of field names used to compare items for duplicates, e.g., "firstName, lastName, email".
• Fuzzy Threshold (required) — Similarity threshold between 0.0 and 1.0; records with similarity above this value are considered duplicates (default 0.8).
• Output Duplicate Info — Boolean indicating whether to include metadata about duplicate removal in the output.
• Options — Additional settings including Case Insensitive Fields (whether field names match ignoring case), Continue On Fail, Debug Mode (include debug info), Skip Unchanged (exclude unchanged items), and Track Changes (include a _changes field with modification details).

Output shape

a list of items with duplicates removed, each item containing the original JSON data possibly augmented by metadata fields if requested

If Output Duplicate Info is enabled, the first item's JSON includes a _deduplicationInfo object summarizing counts and parameters used. If Track Changes is enabled, a _changes field is added with details of the deduplication operation. The deduplication requires at least one field specified in Fields to Check, and the fuzzy threshold must be between 0.0 and 1.0. Items are matched case-insensitively by default unless overridden in options.

Examples

Example 1: Removing duplicate contact records with similar names and emails

Set Fields to Check to "firstName, lastName, email", Fuzzy Threshold to 0.85, enable Output Duplicate Info to get summary metadata, and optionally enable Track Changes for detailed modifications.