Alerts are generated whenever an anomaly is detected, thereby notifying you of any changes in the data. They help maintain system integrity, ensure data accuracy, and facilitate timely decision-making by notifying relevant stakeholders about important changes or potential issues.
Data drift alerts outside the defined threshold limit
- Threshold set to generate alerts: Change in the data properties by 5%.
- Example: Consider a database that has a cardinality value of 50. Let's assume that the cardinality number remains constant for a certain duration and increases by 4% over a period of few days, and the new cardinality value changes to 52. With this condition, the system would not generate an alert as the difference in the value is within the set threshold limit.
- Alert: Consider a scenario in which a value has
been consistently increasing by 4% each day.
- Suppose the cardinality reaches 55 and subsequently decreases to 54 on the following day. Although this decrease is less than 1%; below the established alert threshold, the system generates an alert due to the shift in trend from an upward to a downward direction.
- This example illustrates cases where alerts trigger outside of set thresholds whenever there is a reversal in the direction of the data trend.
Data drift alerts for unique value count
In specific situations, Data Integrity Suite does not generate a data drift alert when the count of unique values changes beyond the established percentage threshold.
- Threshold set to generate alerts: Change in the unique value count by 5%.
- Example: Consider a database containing 100 unique values. If the count of unique values changes by 20%, resulting in a new total of 80 unique records, Data Integrity Suite does not trigger an alert despite the change exceeding the predefined threshold.
- Alert: In this case, it does not trigger an alert
because the number of distinct values exceeds 10.
- The system generates alerts for unique value counts only when the number is less than 10.
- As there are 100 unique values in this scenario, there will be no alert.
- For text fields, however, the threshold is higher, and the number of unique values can be up to 100 before the system considers an alert.
Data drift alerts based on distribution of value count
- Threshold set to generate alerts: Change in the distribution of value count by 3%.
- Example: Consider a database containing 1,000 records. Initially, a specific record appears 10 times within the database. Subsequently, the frequency of this record increases to 60, and the total number of records rises to 1020. In this context, the overall record count has grown by 20 while the occurrences of the specified record have increased by 50.
- Alert: In this scenario, percentage drift is
calculated for a value whose occurrences increased from
10 to 60.
- The proportion changes from (10/1000) × 100 = 1% to (60/1020) × 100 = 5.88%.
- The resulting change in drift percentage is 5.88% - 1% = 4.88%.
- Here, the data drift percentage is 4.88%, which exceeds the preset threshold, resulting in an alert.
Freshness alert
Threshold set to generate alerts: Freshness frequency set to 3 days.
Example: Consider a table that contains multiple records and has not been updated for several days. In this case, the system generates an alert because the condition checks for table updates at a specified frequency, regardless of any modifications or changes to the data.
- The alerts chart reflects a flat line, indicating no notable data changes during this period.
- This alert is not due to variations in the data but rather serves as a notification that the table remains unchanged.
- It serves as a prompt to review and update the data to ensure its ongoing relevance and accuracy.