An observer rule is a part of any observer. You configure observer rules to clearly define when alerts should be generated by essentially defining what you consider a significant change or anomaly and notify observers whenever an anomaly is detected in the selected data assets. The rules can be configured for the different parameters available in the application. Based on the type of alerts generated, there are four different rules supported in the Data Integrity Suite.
- Freshness rule: Alerts are generated when the data fails to update at the expected frequency.
- Volume rule: Alerts are generated whenever there is a change in the number of rows in the data.
- Data Drift rule: Alerts are generated when there is a change in data beyond a specified range.
- Schema Drift rule: Alerts are generated whenever there is a change in a table or column.
Note: Freshness rule cannot be configured for views and is only supported
for tables.
Confidence-based versus threshold-based rules
For freshness, volume, and data drift rules you indicate if you want to create a confidence-based rule or a threshold-based rule. This distinction does not apply to schema drift rules.
- Confidence-based alerts generate alerts based on the certainty that the change meets your criteria for an alert. For example, a freshness generates an alert if a table fails to update at the expected frequency. A confidence-based freshness alert says, only generate that alert if you are 80% confident (as an example) that the table failed to update as expected.
- With threshold-based alerts, you manually assign limits to trigger alerts. For example, you can specifically say that you want to generate an alert if a table fails to update every three days.