Overview
The Generate key step creates a unique identifier for each record, ensuring uniqueness across all datasets. This step reduces the likelihood of duplicate values and enables consistent record matching across different systems and over time.
You can also create a Match Key, which groups records by one or more fields for comparison during matching operations. Records with the same match key are evaluated together, helping to improve both matching efficiency and accuracy.
Key types
The Generate key step allows you to create three types of keys for your pipeline:
- UUID (Universally Unique Identifier)
- Generates a 128-bit random number that's globally unique for each record, ensuring uniqueness across all datasets. This identifier is designed to be created in a decentralized manner.
- Hash
- Generates a unique key for records based on selected fields using the SHA-256 Hash algorithm. This deterministic approach ensures that record IDs remain consistent and repeatable across multiple executions of the pipeline.
- Match Key
- A field or set of fields used to group records for comparison. Records sharing the same match key are evaluated together.
Configuration options
Configure the Generate key step using the following options:
- Step Name
- Defines the name for the step. Provide a meaningful name to help anyone editing steps in a pipeline quickly identify the purpose of the step.
UUID configuration
- Output Field Name
- Assign a unique name to the output field. If a name is not unique, it will result in an error and prevent you from saving the changes.
Hash configuration
- Input Fields
- Select a single value or multiple values from the drop-down for which you want to generate a hash key.
- Output Field Name
- Assign a unique name to the output field. If a name is not unique, it will result in an error and prevent you from saving the changes.
Match Key configuration
- Field
- Select the input field to be used for generating the match key.
- Algorithm
- Select the algorithm used to generate the match key. The available algorithms determine how the input field values are standardized or transformed for grouping purposes.
- Start Position
- Define the starting position from which the value should be considered when generating the match key. This is useful when only a portion of the field value is relevant for matching.
- Length
- Specify the number of characters to include from the selected field, starting from the defined position.
- Output Field Name
- Assign a unique name to the output field. If a name is not unique, it will result in an error and prevent you from saving the changes.
Additional options
- Pick from Template: Select a predefined template to automatically populate field configurations based on common matching patterns.
- Add Another Field: Include additional fields to create a composite match key for more refined grouping.
- Remove Noise Characters: Enable this option to ignore special characters, spaces, or punctuation that may impact matching accuracy.
- Sort Input: Sort the selected input values before generating the match key to ensure consistency, especially when multiple fields are used.
You can preview your changes and choose to Save or Cancel them.