Use target assets to apply a custom rule to selected assets. The rule is evaluated, and scores are generated for linked assets. You can evaluate the rule for fields, fields by condition, or datasets.
Fields
You can choose to evaluate the rule for the selected fields in the target assets section.
- On the Create Rule page, select Select
Fields.What's changed: In the new user experience, the Select Fields appears as Select Target Assets. For more information on the new experience, see About the new user experience.
- Use the search bar to perform a quick search for the required data assets.
- Use available filters to find specific results, and sort data by any parameter displayed in the columns.
- Select the required fields. Fields from multiple
datasets can be selected and added to the rule.Note: It is required to add fields of the same data type for accurate rule evaluation.
- Upon selecting the initial field, the datatype associated with the selected field is automatically updated, and only fields of the same datatype are available for further selection.
- Select Add fields.
- This adds selected fields to the target assets section.
- You can add more fields by selecting Select fields.
- When modifying selected fields, you can choose a new data type to view fields for that data type. Previously selected fields remain part of the rule until you select fields from the new data type.
- After you select fields from a new data type, a popup notifies you that previously selected fields will be removed and the filter will be reset. Click Update to add the newly selected fields to the rule.
- You can remove a field that does not serve the purpose of the rule by clicking the ‘-’ icon next to that field. If you delete a field that is part of the reference dataset, a popup notifies you that another dataset will be set as the reference dataset.
- Datasource type: Select the required datasource type to view only fields or datasets associated with the selected datasource type.
- Datasource: Select the datasource to view the associated fields and datasets.
- Dataset: Select the respective datasets to view the fields associated with that dataset.
- Datatype: Select the required datatype to view only those fields of the associated datatype.
Fields by condition
With this option, the rules can be evaluated for those fields that are selected by configured conditions. This feature helps you select fields that share common attributes.
- Go to the Fields by condition
tab on the Create Rule page within the Target assets
section.What's changed: In the new user experience, on the Create Rule page, select Fields by condition for Target Asset Type. For more information on the new experience, see About the new user experience.
- Select the suitable values from the dropdown. The
dropdown uses the following format:
<attribute><operator><select value> - Select the supported common catalog attributes.
- Dataset: Provides an option to select the datasets
- Datasource: Select the established datasources.
- Datasource type: List the types of datasources added to the application.
- Semantic type: Provides an option to choose fields depending on the selected Semantic type. Whenever the semantic type attribute is selected, only a single value can be selected from the third field.
- Select a suitable operator.
- Select the corresponding values to target the assets for the rule.
- After you define conditions, click Preview Assets to view all fields targeted by the defined condition.
Datasets
To select the datasets that must be a part of the custom rule:
- Go to Datasets on the Create Rule
page and select Select Datasets.Note: In the new user experience, the Select Datasets appears as Select Target Assets.
- Select the target datasets that must be a part of the rule.
- Select Add Datasets to add the
selected datasets to the target assets section.Note: In the new user experience, Add Datasets appears as Select Target Assets.
- You can add more datasets by selecting Select Datasets.
- Datasource type: Select the required datasource type to view only fields or datasets associated with the selected datasource type.
- Datasource: Select the datasource to view only those fields and datasets associated with the respective datasource.
SQL
SQL based quality rules let you define rule conditions directly in SQL queries. You can use existing SQL logic and patterns to build rule expressions. With custom SQL queries, you can define precise conditions to assess quality scores for target datasets and fields. You can also configure required row filters and pass conditions by providing relevant SQL queries.
- Go to SQL and select the specific field or dataset for which you want to apply the SQL rule.
- Click Select Field or Select Dataset. You can sort the data by any column parameter.
- Select the respective field or dataset to associate with the rule, then click Add Field or Add Dataset.
- The selected field or dataset will be added to
Target Assets. Note: When selecting target assets using SQL, you can select a single dataset or field.
- Once the selected field or dataset is added to
Target Assets, configure
the row
filters and pass
condition or fail
condition using SQL queries.Note: When SQL queries are used, row filters and pass conditions are evaluated independently. The Evaluated count shows records matched by row filters, and the Passed count shows records that met the pass condition.
- Test with AI-generated data feature
- Show failed records sample featureWhat's changed: In the new user experience, Show failed records sample appears as Capture Failed Records. For more information on the new experience, see About the new user experience.
- Evaluate rules step within DQ pipelines
- Rule creation using Agentic AI and AI Assist features
Sample SQL rule queries for custom rules
- Define rule logic directly in SQL.
- Handle complex conditions more precisely.
- Reference assets that are not cataloged in Data Integrity Suite.
Example query:
SELECT m.MATNR FROM
dis.dis.MARM m JOIN dis.dis.MARA a ON
a.MATNR=m.MATNR WHERE a.MTART='FERT' AND
m.MEINH='TON'
In this example, several tables are assessed with the JOIN query. These include both cataloged tables available in the Data Integrity Suite's catalog and those tables that are not cataloged. The JOIN query evaluates all of these tables collectively.
Scenario 1: Active customers with valid email
| Row filter use case | Filter only active customers from the dataset. |
| Row filter query |
|
| Row filter description | When applied as a row filter, this condition selects only those customers whose status is marked as Active. |
| Pass condition use case | Customers are required to have a valid email address and the value should not be null. |
| Pass condition query |
|
| Pass condition description | This condition is used to identify which customers have a non-null email address, counting them as passing the rule. |
| Expected outcome | After the rule is evaluated, the row filters are evaluated, and an Evaluated count of assets is generated for active customers and a count of passed records is generated for those customers whose email address is not null. |
Scenario 2: Chicago addresses with CASS certification
| Row filter use case | User addresses must be in Chicago city. |
| Row filter query |
|
| Row filter description | When this condition is set in the row filter, it selects only those customers whose city is listed as Chicago. |
| Pass condition use case | The addresses must be CASS certified. |
| Pass condition query |
|
| Pass condition description | This condition checks which customer records have CASS_CERTIFICATION marked as Y. |
| Expected outcome | After evaluating the rule, an evaluated count is generated representing the customers belonging to the city of Chicago, and a passed count is generated representing those users whose records are CASS certified. |
Scenario 3: Japan customers with at least one order
| Row filter use case | All customers are located in Japan (JPN). |
| Row filter query |
|
| Row filter description | When this query is used in the row filter, it selects only those customers whose country is listed as Japan. |
| Pass condition use case | All customers must have at least one corresponding order. |
| Pass condition query |
|
| Pass condition description | This query, when used as the pass condition, checks that each customer has at least one associated order. |
| Expected outcome | After the rule is evaluated, an evaluated count is generated representing customers who belong to the country of Japan, and a passed count is generated representing those customers who have at least one order. |
Scenario 4: Old invoices with payment records
| Row filter use case | Invoices must be older than 30 days. |
| Row filter query |
|
| Row filter description | This query filters invoices that were created more than 30 days ago. |
| Pass condition use case | Invoices must have an associated payment record. |
| Pass condition query |
|
| Pass condition description | This query identifies invoices that have corresponding payment records. Invoices lacking payment records are treated as failed records. |
| Expected outcome | After rule evaluation, an evaluated count is generated representing the number of records whose invoice is older than 30 days and a passed count is generated representing those records that have associated payment records. |