Candidate Finder retrieves potential matching records from a reference dataset before detailed matching and scoring.
Use the Candidate Finder step to narrow a large reference dataset to a focused set of candidate records. The step uses a lookup key to retrieve records that can then be passed to downstream matching and scoring steps.
Candidate Finder is useful for real-time or transactional matching workflows where comparing an incoming record against every record in a reference dataset would add unnecessary processing. For example, a customer onboarding pipeline can use an incoming field such as First Name, Email, or Customer ID to retrieve a smaller group of possible matches before the matching process evaluates those records in detail.
Key concepts
The Candidate Finder configuration uses these concepts.
| Concept | Description |
|---|---|
| Lookup key | The field used to search the reference dataset. Choose a field with consistent, high-quality values so the step can narrow results effectively. |
| Reference dataset | The known dataset that contains the candidate records. Candidate Finder searches this dataset based on the configured lookup key. |
| Record limit | The maximum number of candidate records returned. The default value is 100, and the maximum value is 500. |
| Field alignment | The mapping between reference dataset fields and incoming dataset fields. Field alignment helps downstream matching steps compare equivalent values. |
Properties
Use the Properties page to select the reference dataset, configure lookup criteria, set the record limit, sort candidate records, and select output fields.
| Field | Description |
|---|---|
| Known dataset | Select the reference dataset that Candidate Finder searches to retrieve candidate records. |
| Incoming field | Select the field from the incoming dataset that is used for candidate lookup. |
| Known field | Select the corresponding field in the reference dataset. The incoming field and known field do not need to have the same name, but they should contain comparable values. |
| Number of Known Records to Return | Enter the maximum number of candidate records returned from the reference dataset. Use the smallest practical value to reduce downstream processing. The default value is 100, and the maximum value is 500. |
| Sort | Optionally sort candidate records before applying the record limit. Sorting helps return the most relevant candidates when multiple records satisfy the lookup criteria. |
| Output Fields | Select fields from the reference dataset to include in the output and align them with fields from the incoming dataset. |
Known record lookup criteria
Known record lookup criteria define how Candidate Finder searches the reference dataset. Select comparable fields from the incoming dataset and the reference dataset. For example, if the incoming record contains FirstName = John, and Incoming field and Known field are both set to FirstName, Candidate Finder retrieves records from the reference dataset where FirstName = John.
Choose lookup fields that are consistently populated and specific enough to reduce the candidate set before matching.
Sort candidates
Use Sort to order candidate records before the record limit is applied. In the Sort Candidates dialog, select the field to sort by and choose Ascending or Descending.
For example, if the reference dataset has three matching customer records with Last Updated values of 2024-01-10, 2024-06-15, and 2024-09-20, sorting by Last Updated in descending order and limiting results to two returns the two most recently updated records.
Output fields
Use Output Fields to select the reference dataset fields that should be included in the output. Candidate Finder automatically maps fields when field names match and data types are compatible. You can modify mappings manually when field names differ or when downstream matching requires a different alignment.
The field selected as the lookup criterion is identified as the lookup key so you can see which field retrieves candidate records.
Considerations
Review these limitations before using Candidate Finder.
- Candidate Finder does not support Amazon S3 as a reference dataset source. Pipelines configured with an S3 reference dataset for Candidate Finder are not executed.
- For real-time service execution in a Databricks environment, the reference dataset must come from a Databricks SQL warehouse or a Databricks-managed dataset.
- In runtime environments such as Databricks or Spark, Candidate Finder processes data by partition. Candidate group numbers are assigned independently within each partition, so ordering or grouping can vary based on partitioning strategy and data distribution.