The Evaluate Rule step allows users to create a step in Data Quality that evaluates a record based on an existing Data Catalog Rule.
It generates new output fields in the dataset that display the results of the rule evaluation for each record. Multiple rules can be chosen, allowing you to either add a field for each rule or aggregate rule results into one field.
Step Name
Defines the name for the step. Provide a meaningful name so that anyone who edits steps in a pipeline can identify the purpose of the step.
Properties
Add Rule
- All Rules: This is the default list of rules, showing all available rules.
- Input Associated Rule: Only the rules relevant to your data are available here.
When selecting the desired rule, it displays the rule definition. Click on Add Rule to include it in the pipeline. The rule is then added to the property panel. You can access the rule details by clicking the info icon.
- Field detected: Black tag
- Field expected but not detected (mapped automatically): Orange tag
- You are informed of the mapping status (Map Fields).
Map Fields
If any field is missing, click on Map Fields. It shows you the Reference Dataset, Data type, and allows you to choose the Input dataset from the drop-down for the missing field. Click on Apply to add the required field.
- If the rule applied is not associated, then select the fields related to the rule. Multiple fields can be selected for evaluation, and filters can be applied to the list of fields by criteria such as data type or semantic type.
- If the rule applied is associated and all the fields have been mapped as required, the transformation step is ready to be executed.
Aggregate Rule
- All True: Only the fields where all individual outputs are true are displayed.
- Any True: Fields where even a single true value is present are displayed.
Anything that is not true or false will be null. The null value indicates that these records/rows were not considered for evaluation in the rule.
Save
Click this button to close the settings window and save the changes made to the transformation settings.
Preview
Click this button to preview the results based on the current transformation settings.
Cancel
Click this button to close the settings window without saving any changes to the transformation settings.