Filter dataset gives you the ability to create alerts based on specific filter conditions on the dataset. You can add single or multiple filter conditions. These conditions are applied to the table fields.You should be familiar with Observer creation steps and have a basic knowledge of filter operators before performing this task.
Observe filtered datasets
You can use filtered datasets to generate volume, data drift, and freshness alerts. Use these steps to filter selected datasets.
- On the main navigation menu, click and select + Create Observer. The Create Observer page appears.
- Select the datasets and fields that you want to observe and click Next.
- Select the Filter Selected Datasets link.
- Select the dataset that you want to filter.
- Select the fields, operator, and type value. You can add more than one
condition. You can choose Match All to match all conditions within the group
(the logical
AND), or you can choose Match Any to match any condition within the group (the logicalOR). - Complete remaining steps of Observer creation.
The Observer is saved with the selected filter conditions, and you are redirected to the Observers page
Considerations when using filter datasets
- When you filter using datetime values, it is assumed that the values are in UTC timezone.
- When dealing with float values, the values you provide are rounded to two decimal places. These rounded values are used for filtering. For example, if you enter a value like 100.2367, it will be rounded to 100.24 for filtering purposes.
- The where clause may have a column that is not in the observed column list for the given table/dataset.
- The column used in the where clause must be a cataloged column.
- If the filter conditions are changed, it will be considered as a change in the data itself, resulting in an increase or decrease in volume, which will generate an alert.
Limitation with quoted database objects
Review your data assets for tables or columns with single or double quotes in their names, like ‘TableName’ or “ColumnName”. Quoted database objects can cause Observer runs to fail. Consider renaming affected objects without quotes.
Workaround
For example, the condition, Firstname equal Joe, where Firstname is the field, equal is the filter condition, and Joe is the value, will generate alerts and profiling results for the table with fields that has a value equal to Joe.
The profiling statistics are also generated based on the filtered set only and not the entire dataset. For example, a table that has a total of 100 records (without filter) displays only 10 records, in this case, all statistics are calculated based on the filtered records (10 in this case).