Duplicate Detection

Fuzzy matching using Levenshtein distance to find near-duplicates.

How It Works#

Our duplicate detection uses the Levenshtein distance algorithm to calculate similarity between text values. This catches variations that exact matching misses:

Typo

"John Smith"

vs "John Smth"

1 character difference
Suffix

"ABC Company"

vs "ABC Company Inc"

Extra suffix added
Abbreviation

"123 Main St"

vs "123 Main Street"

Expanded abbreviation

Configuration#

Similarity Threshold

Set the minimum similarity percentage (0-100%) for records to be considered duplicates. Default is 85%.

Threshold85%
Loose (more matches)Strict (fewer matches)

Column Weights

Prioritize certain columns when calculating overall similarity:

PriorityColumn PatternsWeight
Highname, title, company, email1.0
Mediumaddress, phone, city, url0.75
LowEverything else0.5

Merge Strategies#

When duplicates are found, choose how to combine them:

first

Keep the first non-empty value

Best for data where earlier records are more authoritative

longest

Keep the longest value

Best for fields where more detail is better (descriptions, addresses)

most_complete

Prefer non-empty, then longest

Best general-purpose option for most datasets

Note
Duplicate detection runs after normalization, so "555-123-4567" and "(555) 123-4567" will be recognized as the same phone number.
PPC TransformationsJSON-LD Basics