This step matches records from one or more datasets based on defined match rules and scoring algorithms, identifying potential duplicates or related records.
The Match Records step identifies and groups records that represent the same entity based on configurable match rules and scoring algorithms. This step is essential for data quality tasks such as de-duplication, record linking, and data consolidation.
Matching approaches
You can match records in the following ways:
-
Within Same Dataset: Compares records within a single dataset to identify duplicates or related records. This is useful for de-duplication and identifying records that represent the same entity.
-
Across Two or more Datasets: Matches records across multiple datasets in a pipeline flow, allowing you to link data from different sources as they move through the pipeline. Interflow matching can also be used with a single dataset to identify and link related records throughout the pipeline.
Common use cases
- Identifying duplicate customer records in a database
- Linking records from different data sources that represent the same person or business
- Finding records with similar addresses to consolidate household information
- Matching contact information across multiple datasets for data enrichment
- Detecting near-duplicate records with minor spelling variations or formatting differences
Configuration overview
The Match Records step configuration consists of the following main components:
Match Pattern
You can choose between two types of match patterns:
-
Compare all records (Intraflow): Each record is compared with others in the same match key group. A Match Rule must be set up to specify the Match key.
-
Compare with reference (Interflow): Compares source records to reference records (all records except source records are treated as reference) within the same match key group. To get started, you need to define the following:
- Incoming source field: Specifies the field used to identify source records for reference-based matching.
- Incoming record value: The value in the source field used to identify source or incoming records.
Match Rule
Select fields and algorithms to set up the match rule.
Settings
Use the Settings section to define how unmatched source records and reference record comparisons are handled during the matching process.
- Assign GroupID 0 to unmatched source records: Assigns a default GroupID value of 0 to source records that do not find a match.
- Assign unique GroupID to unmatched source records: Assigns a unique GroupID to each unmatched source record.
Reference comparison settings
- Compare all reference records: Compares each source record against all eligible reference records within the same match key group.
- Return unmatched reference records: Includes reference records that did not match with any source record in the output.
- Stop comparing on finding duplicates: Stops further comparison after the specified number of duplicate matches is found for a source record.
Match key
A match key is a field or combination of fields that groups records for matching. Records with the same match key are compared against each other. This is essential for blocking strategy, which improves performance by reducing the number of comparisons needed. You can create a Match Key using the Generate key step, or you can use a key that already exists in your dataset. For instructions on generating a Match Key for your data quality pipeline, see the Generate Key documentation.
Union output field
When the Union option is enabled in the Match and Group pipeline, the system creates
a new output field named Union_Source_Identifier. This field is
used as the Matching as Source field to identify the source dataset or input record
contributing to the union output. The Union_Source_Identifier field
helps distinguish records originating from different inputs during matching and
grouping operations, especially in multi-input pipeline configurations.
Match scenarios
A match scenario defines a specific set of fields and algorithms used to match records. You can define multiple scenarios and combine them with logical operators (AND, OR) to create complex matching logic.
Creating a match scenario
- Click Add to create a new match scenario.
- Enter a descriptive name for the scenario.
- Click Match Scenario Options to configure the scenario settings.
- Add fields to the scenario by clicking Add in the field table.
- Configure the scoring method, algorithms, and threshold for each field.
Match scenario options
Name
The name of the match scenario. Use a descriptive name that indicates the fields or criteria being matched.
Matching Method
Specifies how field scores are combined to determine if records match:
- All true: All fields must meet their threshold scores for records to match.
- Any true: At least one field must meet its threshold score for records to match.
- Based on threshold: The overall match score must meet a specified threshold.
Missing Data
Specifies how to handle missing or blank values:
- Ignore blanks: Blank values are not considered in the matching calculation.
- Compare blanks: Blank values are compared during matching and contribute to the match score based on the comparison result.
- Count as 100: Missing or blank values are treated as a perfect match and contribute a score of 100 to the matching calculation.
- Count as 0: Missing or blank values are treated as a non-match and contribute a score of 0 to the matching calculation.
Scoring Method
Specifies how individual field scores are calculated:
- Average: The average of all algorithm scores for the field.
- Maximum: The highest score from all algorithms applied to the field.
- Minimum: The lowest score from all algorithms applied to the field.
Children
Displays the fields included in this match scenario. You can expand or collapse this section to view the fields being used for matching.
Adding fields to a match scenario
- In the Match Scenario configuration, click Add to add a field.
- Select a field from the available fields list.
- Configure the following for each field:
- Score method: Choose how to calculate the field score (Average, Maximum, or Minimum).
- Algorithms: Select one or more matching algorithms to apply to the field.
- Threshold: Set the minimum score (0–100%) required for the field to be considered a match.
- Click the action button to remove or modify the field configuration.
Matching algorithms
Matching algorithms compare field values and calculate a similarity score. You can apply multiple algorithms to a single field, and the scores are combined based on the selected scoring method.
| Algorithm | Category | Description |
|---|---|---|
| Acronym | String | Determines whether a business name matches its acronym by looking for acronym data, else it creates an acronym using the first character of every word. |
| Character Frequency | String | Determines the frequency of occurrence of each character in a string and compares the overall frequencies between two strings. |
| Consonant | Exact |
Only consonants are compared. Vowels are removed from the comparison. It returns a match if consonants from two values match exactly. Results may not be accurate if the data contains multibyte characters.
|
| Daitch-Mokotoff Soundex | Phonetic | Phonetic algorithm that allows greater accuracy in matching of Slavic and Yiddish surnames with similar pronunciation but differences in spelling. Coded names are six digits long, and multiple possible encodings can be returned for a single name. This option was developed to respond to limitations of Soundex in the processing of Germanic or Slavic surnames. |
| Date | Date |
Compare date fields regardless of the date format in the input records. Click Edit in the Options column to specify the following:
|
| Double Metaphone | Fuzzy |
Determines the similarity between two strings based on a phonetic representation of their characters. Double Metaphone is an improved version of the Metaphone algorithm, and attempts to account for the many irregularities found in different languages. Metaphone3 improves upon this algorithm. |
| Edit Distance | Similarity and Distance | Determines the similarity between two strings based on the number of deletions, insertions, or substitutions required to transform one string into another. |
| Euclidean Distance | Similarity and Distance | Provides a similarity measure between two strings using the vector space of combined terms as the dimensions. It also determines the greatest common divisor of two integers. It takes a pair of positive integers and forms a new pair that consists of the smaller number and the difference between the larger and smaller numbers. The process repeats until the numbers are equal. That number then is the greatest common divisor of the original pair. For example, 21 is the greatest common divisor of 252 and 105: (252 = 12 × 21; 105 = 5 × 21); since 252 − 105 = (12 − 5) × 21 = 147, the GCD of 147 and 105 is also 21. |
| Exact match | String | Determines if two strings are the same. |
| Initials | String | Used to match initials for parsed personal names. |
| Jaro-Winkler Distance | Similarity and Distance | Determines the similarity between two strings based on the number of character replacements it takes to transform one string into another. This option was developed for short strings, such as personal names. |
| Keyboard Distance | Similarity and Distance | Determines the similarity between two strings based on the number of deletions, insertions, or substitutions required to transform one string to the other, weighted by the position of the keys on the keyboard. Click Edit in the Options column to specify the type of keyboard you are using: QWERTY (U.S.), QWERTZ (Austria and Germany), or AZERTY (France). |
| Koeln | Phonetic | Indexes names by sound as they are pronounced in German. Allows names with the same pronunciation to be encoded to the same representation so that they can be matched, despite minor differences in spelling. The result is always a sequence of numbers; special characters and white spaces are ignored. This option was developed to respond to limitations of Soundex. |
| Kullback-Leibler Distance | Similarity and Distance | Determines the similarity between two strings based on the differences between the — of words in the two strings. |
| Metaphone | Fuzzy |
Determines the similarity between two English-language strings based on a phonetic representation of their characters. This option was developed to respond to limitations of Soundex. |
| Metaphone (Spanish) | Phonetic | It determines the similarity between two strings based on a phonetic representation of their characters. This option was developed to respond to the limitations of Soundex. |
| Metaphone 3 | Fuzzy | Improves upon the Metaphone and Double Metaphone algorithms with more exact consonant and internal vowel settings that allow you to produce words or names more or less closely matched to search terms on a phonetic basis. Metaphone 3 increases the accuracy of phonetic encoding to 98% by allowing for differences in spelling due to dialects or pronunciation. This algorithm was developed to respond to limitations of Soundex. |
| NGram Distance | Similarity and Distance | Calculates in text or speech the probability of the next term based on the previous n terms, which can include phonemes, syllables, letters, words, or base pairs and can consist of any combination of letters. This algorithm includes an option to enter the size of the NGram; the default is 2. |
| NGram Similarity | Similarity and Distance |
Determines similarity between two strings based on the length of the longest common subsequence of phonemes, syllables, letters, words or base pairs. The algorithm includes the following options:
|
| Numeric String | String |
Compares address lines by separating the numerical attributes of an address line from the characters. For example, in the string address 1234 Main Street Apt 567, the numerical attributes of the string (1234567) are parsed and handled differently from the remaining string value (Main Street Apt). The algorithm first matches numeric data in the string with the numeric algorithm. If the numeric data match is 100, the alphabetic data is matched using Edit distance and Character Frequency. The final match score is calculated as follows: |
| Spanish Metaphone | Fuzzy | Determines the similarity between two Spanish-language strings based on a phonetic representation of their characters. This option was developed to respond to limitations of Soundex. |
| Nysiis | Phonetic | Phonetic code algorithm that matches an
approximate pronunciation to an exact spelling and indexes words
that are pronounced similarly. Part of the New York State
Identification and Intelligence System. Say, for example, that you are looking for someone's information in a database of people. You believe that the person's name sounds like "John Smith", but it is in fact spelled "Jon Smath". If you conducted a search looking for an exact match for "John Smith" no results would be returned. However, if you index the database using the NYSIIS algorithm and search using the NYSIIS algorithm again, the correct match will be returned because both "John Smith" and "Jon Smath" are indexed as "JANSNATH" by the algorithm. This option was developed to respond to limitations of Soundex; it handles some multicharacter n-grams and maintains relative vowel positioning, whereas Soundex does not. Note: This algorithm does not process non-alpha
characters; records containing them will fail during
processing.
|
| Phonix | Phonetic |
The Phonix algorithm is a Soundex variant. While the Soundex phonetic property is restricted to the collection of similar sounding consonants into different classes, the algorithm for computing the Phonix codes uses elaborate substitution rules Preprocesses name strings by applying more than 100 rules to single characters or to sequences of several characters. 19 of those rules are applied only if the character or characters are at the beginning of the string, while 12 of the rules are applied only if they are at the middle of the string, and 28 of the rules are applied only if they are at the end of the string. The name string is encoded into a code that is comprised by a starting letter followed by three digits (removing zeros and duplicate numbers). This is more sophisticated than Soundex. It is also more complex and therefore slower than Soundex. |
| Sonnex | Phonetic | Determines similarity for two
French-language strings based on a phonetic representation of their
characters. It returns a Sonnex coded key of the selected fields. |
| Soundex | Phonetic |
Determines the similarity of two strings based on a phonetic representation of their characters. This is less sophisticated than Phonix. |
| SubString | Exact |
Compares two values based on a substring within the values. This determining the similarity based on a substring, particularly where those values contain long strings of characters, or many words.
|
| Syllable Alignment | Phonetic |
Combines phonetic information with edit distance based calculations. Converts the strings to be compared into their corresponding sequences of syllables and calculates the number of edits required to convert one sequence of syllables to the other. |
Selecting algorithms
Choose algorithms based on the type of data and the matching requirements:
- For names and text fields with potential spelling variations, use Jaro-Winkler or Edit Distance.
- For standardized codes or identifiers, use Exact Match.
- For fields where character composition is important, use Character Frequency.
- Combine multiple algorithms to increase matching accuracy.
Threshold score
The threshold score is the minimum match score (0–100%) required for a field or record to be considered a match. Set thresholds based on your data quality requirements:
-
High threshold (80–100%): Use for critical fields where accuracy is paramount. Reduces false positives but may miss some valid matches.
-
Medium threshold (60–80%): Balanced approach for general matching scenarios.
-
Low threshold (0–60%): Use when you want to capture more potential matches but be prepared for more false positives.