Configure rule schedules, view rule scores and definitions, monitor rule run status, and examine failed records to manage quality rules.
Quality gives you control over rule runs and visibility into rule performance. You can configure schedules to reduce processing costs, view scores and definitions, monitor run status, and examine failed records to troubleshoot quality issues.
Configure rule schedule
Manage rule run schedules directly from the rules list. Turn rule runs off or on to reduce processing costs while keeping access to historical scores and run details.
You can:
- Turn off a rule schedule directly from the rules list, preventing the rule from running on the catalog schedule setup during onboarding.
- Turn on a previously turned off rule schedule, allowing it to resume running.
- Access prior scores and the details of how those scores were calculated, even when a rule schedule is turned off.
- Resume scoring when you turn on a rule schedule.
- Manually run a disabled rule as needed, giving you full control over rule runs.
- Go to .
- Open the context menu next to the associated rule.
- Select Disable Scheduling to prevent the rule from running on the configured schedule. The scheduling status updates on the Rules page.
- To turn scheduling on again, open the context menu and select Enable Scheduling. The rule resumes running on the configured schedule.
View rule scores
Quality scores evaluate dataset quality and reliability by using factors such as accuracy, completeness, consistency, timeliness, validity, uniqueness, and relevance. View scores for each field or dataset linked to a rule after successful run.
View quality score trends
View generated quality score trends to see how rule performance metrics change over time.
To view generated quality score trends:
- Go to Quality and open the Rules page.
- Select the custom rule for which you want to view score trends.
- On the Rule Scores page, select the trend icon next to the field score for which a quality score is generated.
This displays a graph with relevant details to help you visualize score trends over time.
View rule definition
A rule definition specifies the conditions and parameters used to evaluate the rule. Review all configuration details, including dimensions, scoring bands, pass conditions, and expressions.
View rule run status
The rule run status indicates the current state of rule runs for all rules in Quality. Track progress, identify errors, and troubleshoot rule run failures.
Rule run status helps you track rule run progress and effectiveness. You can identify errors or failures that occur during rule runs and troubleshoot them.
Depending on the rule run status, the following statuses are displayed for rule evaluation in Quality:
- Completed: Indicates that the rule was successfully evaluated.
- In progress: Indicates that the process of rule evaluation is initiated.
- Partially completed: Indicates that rule was partially evaluated.
- Failed: Indicates that the rule evaluation failed.
Completed
The completed status means all datasets or fields in the rule were evaluated successfully with no exceptions. Select the run status to view evaluated fields or datasets with the following details:
- Dataset or Field: Name of the datasets or fields considered for rule evaluation.
- Score: Score generated for each of the datasets or fields.
- Last evaluated: Timestamp when the rule was last evaluated.
In progress
In progress means the rule run started but is not complete. Select the rule run status to view current status and associated dataset or field details.
- Completed: The completed page displays the details of the fields that were evaluated successfully.
- Failed: Displays a count of the fields that were not evaluated along with an error summary of all the associated datasets or fields. All errors are grouped by datasets. Select View details to see details on all the associated datasets or fields.
- In-progress: Displays a count of fields or datasets that are still under evaluation along with their details. All fields are grouped by datasets on this page.
Partially completed
Partially completed means some datasets or fields were evaluated successfully and others failed during the rule run. Select the run status to view details and the most recent evaluation timestamp.
The partially completed run status displays the details on two pages:
- Completed: The completed page displays the number of datasets or fields that were evaluated successfully out of the total number of available assets. The details include the name of the dataset or field, the generated score, and the timestamp when the dataset or field was last evaluated.
- Failed: The failed page displays the number of datasets or fields that failed during rule evaluation. The error summary provides details on all the errors generated. Select View details in the error to display the cause for failure along with the details of the associated dataset and the connection.
Failed
The failed status means none of the rule datasets or fields were evaluated successfully. Select the failed status on the Rules page to view run details and the last evaluation date.
- It displays a count of the number of fields or datasets that were not evaluated during rule run.
- The error summary displays all the errors generated along with the details of the associated datasets or fields.
- Select View details to see the cause of rule failure and details of the associated dataset and catalog in the error details.
Common error messages
When a rule evaluation fails, an associated error message appears alongside the rule run status. This message explains why the rule evaluation did not succeed. For more information about common rule run failure messages, see Troubleshoot quality errors.
View failed records
Failed record samples show records that failed rule evaluation. For SQL rules, failed records can also be persisted when failed records storage and failed-record output logging are turned on.
Failed record samples can be viewed only for custom expression rules created in Quality. To persist failed records for an SQL rule run, use Output failed records.
Before you turn on the failed records toggle on the create rule page, you must turn on the failed records sample toggle on the configuration page.
To enable failed records sampling:
- Go to .
- Turn on the Failed Records Sample toggle.
- Select Save.
When failed records sampling is available on the Configuration page, you can turn on the Show failed records sample toggle on the Create Rule page. Failed record details are generated only for rules run with this toggle turned on.
Configure failed records storage
Use failed records storage to persist failed records generated by SQL quality rule runs.
Configure failed records storage before you turn on Output failed records for SQL rules. When storage is turned on and an output target is configured, failed records from SQL rule runs are persisted to the configured target.
- Go to .
- Turn on Persist a log of all failed records for each rule run.
- Under Outputs, enter an output name.
- Choose the target option and target dataset.
- Choose the connection associated with the target dataset.
- Select Save.
The failed records output includes these fields:
| Field | Description |
|---|---|
record_content |
Content of the record that failed rule evaluation. |
rule_id |
Identifier of the quality rule that generated the failed record. |
rule_name |
Name of the quality rule that generated the failed record. |
asset_id |
Identifier of the technical asset associated with the failed record. |
asset_path |
Path of the technical asset associated with the failed record. |
target_type |
Type of target associated with the rule evaluation. |
run_timestamp |
Date and time when the rule was evaluated and the failed record was generated. |