Custom rules in Data Integrity Suite provides the flexibility to selectively choose the fields or datasets that must be a part of the associated rule and configure appropriate conditions so that the rule evaluates the parameters accordingly.
Create custom rules
To create a custom rule:
- Navigate to .
- Click Create Rule. This will open the create rule page
where you can enter the required details.
- Rule Name: Provide a suitable name to the rule. This would appear in the rules page and can be used for easy identification. For example, if you are creating a rule to check the assets in a Snowflake connection for the completeness dimension, you can name the rule as Snowflake_Completeness.
- Description: Provide an optional description to the rule. This section can be used to provide more details on the rule. For example, if a rule is created to test for completeness in the datasets belonging to a Snowflake connection, the description can be "The rule is created to test the completeness of the datasets and schemas in the Snowflake connection".
- Dimension: Select an appropriate dimension to be associated with the rule. These dimensions are used only for tagging purpose. Selecting a dimension does not have an impact on rule evaluation. For example, if a rule is created to check for the presence of records, then the completeness dimension can be selected for the rule. If a rule is created to check for uniformity of information across data assets, then the consistency dimension can be selected for the rule.
- Target assets: Select the target assets using any
of the following options.
- Fields: Select the fields that have to be associated with the rule.
- Fields by condition: With this option, you can define a condition, and the fields that satisfy the condition are automatically considered for rule evaluation. With this option you can select fields across datasources with common attributes.
- Dataset: Select the datasets that have to be associated with the rule.
- Add Evaluated Rows Filter: Target the rules to be
evaluated for those datasets or fields that satisfy the user defined
conditions. When you are targeting fields by conditions, you can define the conditions in the row filter and pass conditions by using a value alias component.Note: Configuring row filters is optional. The required row filters can be added depending on your specific requirements or preferences.
- Pass Conditions: Once the appropriate fields are
selected, proceed to select the conditions that have to be satisfied for
the rule to be evaluated. Note: If the pass condition is being applied to a dataset, it is required to map all the fields in the dataset post which the rule can be created. A rule cannot be saved without all the fields being mapped.
On enabling the show failed records sample toggle, details of failed records will be generated after the corresponding rule run. This would help identify the cause of failure during rule evaluation. You need to enable the Show failed records sample option at the workspace level in the Configuration tab before you can use this feature while creating a rule.
On turning the toggle on for enable scheduling, it schedules the rule to run on the catalog schedule that is configured while establishing a new data connection.
-
Once all the parameters are configured, click Create. The created custom rule would appear in the rules page.
NULL values. This behavior is
observed for the following basic functions: Not equal, Less than, Less than or
equal. <Value><is not null> . This would exclude null records for
evaluation and the score would be generated accordingly.You can configure the custom rules from two different locations in the application:
- The rules page
- The rule details page
The rules page
There are several actions that can be performed with the custom rule from the rules page.
Run a custom ruleYou can run a custom rule instantly whenever it must be evaluated.
- Navigate to .
- Click the ellipsis next to the relevant rule and select Run. This starts the rule evaluation process and creates a job for this process. You can find the job in the Assessments* Limited Availability tab, located within the Jobs tab in the Quality section.
- When the run is initiated, the status of rule execution is updated to In Progress in the Rules page.
- The status of the rule evaluation is updated depending on either of the end
results:
- Completed: Indicates that the rule was successfully evaluated.
- In Progress: Indicates that the process of rule evaluation is initiated.
- Partially completed: Indicates that rule was partially evaluated.
- Failed: Indicates that the rule evaluation failed.
- Stopped: Indicates that the rule execution was stopped.
Edit custom rule
- Navigate to .
- Click the ellipsis next to the associated rule and click Edit. This will open the Edit Rule page where you can edit the configured parameters.
- Once the parameters are configured, click Save.
Duplicate custom rule
You can duplicate an existing rule if you would want to create a new rule similar to an existing rule. This would eliminate the process of creating a custom rule from scratch.
- Navigate to .
- Click the ellipsis next to the associated rule and click Duplicate.
- This will open the Create Rule page where you can modify the configured parameters and click Create .
Delete custom rule
When an existing custom rule in the Data Integrity Suite does not meet the requirements, you can delete the rule from the application.
- Navigate to .
- Click the ellipsis next to the associated rule and click Delete. This will display the Delete Rule dialog to confirm if you would want to delete the rule.
- Click Delete in the dialog. This will permanently delete the rule from the application.
The rule details page
You can configure the created custom rule form the rule details page.
- Navigate to .
- Click the corresponding rule name which will navigate you to the rule details page.
- Within the rule details page, you can modify the following parameters:
- Edit: Click Edit to modify the rule details and parameters.
- Run: Click the ellipsis located at the top right corner of the rule details page and click Run. This will initiate the rule evaluation process, and the status is updated to In Progress in the Rules page.
- Duplicate: Click the ellipsis located at the top right corner of the rule details page and select Duplicate. This will open the Create Quality Rule page where you can modify the configured parameters. Once the parameters in the rule are modified, Click Create.
- Enable/Disable scheduling: Click the ellipsis located at the top right corner of the rule details page and click Enable or Disable scheduling of rule execution. This will enable or disable the scheduling of rule execution depending on the value selected.
- Delete: Click the ellipsis icon located at the top right corner of the screen in the rule details page and Click Delete.