An alert is generated depending on the configured observer rules. When you create an Observer, you define specific rules that monitor your data assets for unexpected changes. The system evaluates these rules during each profile run associated with an Observer and generates alerts when anomalies are detected based on the thresholds and conditions you have configured.
- Alert type: The system can generate four types of alerts based on your observer rules: Freshness, Volume, Data drift, and Schema drift.
- Alert level: Each alert is classified as either Critical or Warning based on the severity level configured in your observer rules.
- Confidence percentage: The system calculates a confidence level for each alert, indicating the likelihood that the detected change represents a genuine anomaly.
- Significance thresholds: You can configure minimum change values that must be exceeded before an alert is triggered, helping to reduce false positives and ensure only meaningful changes generate alerts. The alert generation compares current profile run data against historical patterns and previous runs to identify deviations that exceed your configured thresholds.
Confidence-based alert generation scenario
When you configure a confidence-based rule in the Observer, alerts are generated only after five successful observer runs and the profiling statistics collected from these runs.
Freshness alert
For confidence-based Freshness alerts, the system analyzes five consecutive profiles before generating an alert. The alert is triggered based on the table's update history. If the table is not updated during the observer runs, an alert will be generated.
Conversely, if the table updates successfully with each observer run, no alert will be triggered.
Volume alert
For confidence-based Volume alerts, the system analyzes five consecutive profiles before generating an alert. The alert is triggered by comparing the latest run with the previous run. If the change in value exceeds the minimum threshold set for alert generation, an alert is generated based on the comparison between the previous run and the current run. After the observer run completes, either a warning or critical alert will be generated.
Data drift alert
For confidence-based Data drift alerts, the system analyzes five consecutive profiles before generating an alert. The alert is triggered by comparing the latest run with the previous run. If the change in value exceeds the minimum threshold set for alert generation, an alert is generated based on the comparison between the previous run and the current run. For textual values, the minimum change is specified as the number of characters, while for numerical data, it is specified as a percentage change. You can configure the alert levels accordingly, and after the observer run completes, either a warning or critical alert will be generated.
Threshold-based alert generation scenario
When you configure a threshold-based rule, an alert is generated as soon as the set threshold is breached. You can configure threshold-based alerts for Freshness and Volume rules.
Freshness rule
When you set a threshold for a specific frequency, the observer checks if the table has updated according to the set frequency and instantly generates alerts when the table is not updated. For example, if the Freshness Frequency is set to Every 2 days, the Observer checks if the table is updated every 2 days. If the table is not updated, it immediately triggers an alert.
Volume rule
With threshold-based alerts in the Volume rule, you can configure the number of runs required to trigger an alert using a moving average. The moving average generates alerts based on the average change in data over time, considering the number of profile runs specified in the data points. You can also choose the last profile run option to generate alerts based solely on the most recent profile run.
Additionally, you can configure the range to generate warning and critical alerts within the set threshold.