Quality rules are criteria and constraints that help ensure data accuracy, completeness, reliability, and relevance. Rules validate and cleanse data, enforce standards, and help prevent data errors.
Data Integrity Suite supports two rule types.
- Expression: Rule evaluation conditions are defined by using the Precisely expression builder.
-
SQL:
Rule evaluation conditions are defined by using SQL queries.
Tip:
- Rules created by default in the application are marked with a default tag.
- Default rules are automatically created in the workspace and are triggered on all assets and fields.
Default rules run on all assets for the respective datasource when Run Quality Rules and Calculate Scores toggle is turned on on the Insights page while establishing a new datasource. This evaluates assets for data distribution, completeness, and validity, and generates a Quality Score. Rules assess data completeness, consistency, accuracy, and timeliness. Organizations can use predefined rules to identify and address data issues, errors, and inconsistencies systematically.
Profiling in Data Integrity Suite includes categories that provide insights into data characteristics. Each category displays metrics, including counts and percentages of valid, invalid, and null entries, with visuals such as bar charts for data distribution and value counts. Profiling statistics are generated for datasets and fields when the Profile Datasets toggle is turned on on the Insights page while establishing a new datasource.
Go to to view the rule list.
Use Create Rule to create a rule, use the search box to find a rule by name, or select Refresh to update the rule list.
-
Rule
: Displays the rule name. Select it to view details, including:
- Rule Scores: Displays the list of associated fields, each rule score, and the date when the rule last ran. Rule score is a quantitative quality measure based on the rule criteria.
- Rule Definition: Provides the rule description, dimension, scoring bands, total number of associated target fields, pass conditions, result type, and associated expression.
-
Dimension
: Shows the rule dimension. Each dimension represents a quality criterion that data must meet. Rules evaluate quality in that dimension.. These dimensions provide a framework for evaluating characteristics such as
accuracy
,
completeness
,
consistency
,
timeliness
,
validity
,
relevance
,
uniqueness
, and
integrity
. By analyzing quality across these dimensions, you can identify areas for improvement and implement strategies to enhance the overall quality and reliability of your data assets.
Note: These dimensions are only used for tagging, which helps you identify the dimension on which the rule is evaluated. These dimensions do not impact rule evaluation.
-
Targets
: Shows the number of assets targeted by the rule.
Tip: After Data Integrity Suite evaluates a quality rule successfully, it displays the number of assets the rule evaluated. If you remove any assets from the associated connection, the targets area on the Rules page automatically updates to show the current number of assets the rule targets. If the rule has no targeted assets available for evaluation, the target assets area displays no entries and appears blank.
-
Rule Type
: Displays the rule type
- Expression: Indicates that the conditions for rule evaluation are defined using the Precisely expression builder.
-
Sql:
Indicates that rule evaluation conditions are defined by using SQL queries.
Tip: Rules created by default in the application are marked with a default tag.
- Run Status: Displays the status of rule execution.
- Last Evaluated: Displays the timestamp when the rule was last evaluated.
- Scheduling: Displays if the scheduling for the rule is enabled or disabled.
- Description: Shows a detailed description of the rule.
Default rules in Data Integrity Suite
-
Default Blank Check
: The rule is applied to
String-type
fields to check for blank values in records.
Warning: The Default Validity Check rule, upon successful evaluation, generates only a quality score. However, the total count of evaluated records is not displayed for the Default Validity Check rule.
- Default Null Check: The rule is applied to all records in a field to check for null values.
- Default Validity Check: Checks if a field contains valid data according to the semantic type.
After successful evaluation, these rules generate a quality score for associated fields. Open the context menu next to the associated rule to manually initiate the rule run or configure scheduling.
SQL rules
- Use SQL syntax and functions supported by the selected datasource.
- Database-specific functions and operators might not be supported across all datasources.
- Unsupported SQL syntax or features can cause rule validation or execution to fail.
- The SQL examples in this topic are provided for illustration and might require modification to match your datasource.
For example, this query can fail because both selected columns are returned as Name:
SELECT a3.Name, a.Name FROM hive_metastore.basicprofiling.alldatatypes3 a3 LEFT JOIN hive_metastore.basicprofiling.alldatatypes a ON a3.Id = a.Id WHERE a3.IsActive = trueAssign an alias to one of the duplicate columns:
SELECT a3.Name AS Name3, a.Name FROM hive_metastore.basicprofiling.alldatatypes3 a3 LEFT JOIN hive_metastore.basicprofiling.alldatatypes a ON a3.Id = a.Id WHERE a3.IsActive = trueConfigure custom quality rules
Create custom rules to evaluate specific fields or datasets against conditions you define.
To create a custom rule:
- Go to .
-
Select
Create Rule
to open the Create Rule page.
- Rule Name: Enter a descriptive rule name. This name appears on the rules page for easy identification. Example: Snowflake_Completeness.
- Description (optional) : Optionally, enter a description that explains the rule's purpose. Example: "Tests the completeness of datasets and schemas in the Snowflake connection."
- Dimension: Select a dimension to tag the rule. Dimensions are for organization only and do not affect rule evaluation. Example: Select Completeness to check for record presence, or Consistency to check for uniformity across assets.
-
Target Asset Type
: Select the assets to evaluate using one of these options:
- Fields: Select specific fields to associate with the rule.
- Fields by condition: Define a condition to automatically include fields that match it. This lets you select fields across datasources with common attributes.
- Dataset: Select specific datasets to associate with the rule.
-
Add Row Filters
: Optionally, define conditions to limit rule evaluation to specific datasets or fields.
When targeting fields by conditions, define row filter conditions by using a value alias component.
Note: Row filters are optional. Add them based on your specific requirements. -
Pass Conditions
: Select the conditions the rule must satisfy during evaluation.
Note: When applying pass conditions to a dataset, you must map all fields in the dataset before saving the rule.
-
Capture Failed Records: Enable this toggle to generate failed-record details after rule execution. This helps identify failure causes. First, enable this option at the workspace level on the Configuration page.
-
Output failed records: Turn on this toggle to persist failed records from the rule run to the configured failed-records output. This option is available for SQL rules when failed records storage is turned on and an output target is configured.
-
Enable Auto-Run Schedule: Enable this toggle to schedule the rule to run on the catalog schedule configured when you established the data connection.
-
Select Create. The custom rule appears on the rules page.
- When using the Not equal, Less than, or Less than or equal functions, null records from the database are included in score calculations.
- To exclude null records from evaluation, add a second condition:
<Value><is not null>. This generates scores without null values.
- Edit: Select Edit to modify rule details and parameters.
- Run: Open the context menu, and then select Run to start rule evaluation. The status updates to In Progress on the Rules page.
- Duplicate: Open the context menu and select Duplicate. The Create Quality Rule page opens. Modify the parameters and select Create.
- Enable or Disable scheduling: Open the context menu, and then select Enable or Disable to control rule execution scheduling.
-
Delete:
Open the context menu and select
Delete
.
Tip:
- Rule execution status syncs with assessment job status. If an assessment job fails, the corresponding rule status also shows as failed.
- When you edit a rule, if any target assets have been deleted, the rule automatically removes them and resets the pass conditions.