Configure data drift alerts - Precisely Data Integrity Suite

Configure data drift alerts to track unexpected changes in data statistics such as cardinality, min or max values, and completeness. Data drift rules generate alerts when selected statistics increase or decrease beyond your expected thresholds.

1. Go to Observability > Observers and select + Create Observer.
2. On the Create Observer page, select your data assets and click Next.
3. In the Data Drift row, click the gear icon in the Configure column.
4. Select the statistics for the data types you want to observe.

   All Data Types:

   • Completeness: The degree to which a table or column contains all expected or required values. It is represented as a percentage. For example, a complete table with 1000 expected rows would have all 1000 rows present with no missing or null values. Similarly, if a column is expected to have a particular type of data (such as numerical values), a complete column must have all values of the expected type.
   • Unique value count (Cardinality): A measure of the number of distinct elements in a dataset. For example, in a dataset with names {Mark, John, Ashley, Jonas, James, Emma}, the cardinality is 6.

   Numeric Data:

   • Minimum: The lowest value in the dataset. For example, in the set {1215, 2000, 5263, 1126, 3687}, the minimum is 1126.
   • Maximum: The highest value in the dataset. For example, in the set {1215, 2000, 5263, 1126, 3687}, the maximum is 5263.
   • Mean: The average of all elements, calculated as: (Sum of all elements) / (Number of elements). For example, {1215, 2000, 5263, 1126, 3687} has a mean of (1215+2000+5263+1126+3687)/5 = 2658.2.
   • Standard deviation: A measure of variation in data points from the mean, calculated as the square root of variance. Higher values indicate data points are spread out from the mean; lower values indicate they are close to the mean.

   Textual Data:

   • Minimum length: The smallest length of text in the dataset. For example, in {Sao Paulo, Mexico, Tokyo, Shanghai, Cairo, Mumbai}, the minimum length is 5 (Cairo, Tokyo).
   • Maximum length: The largest length of text in the dataset. For example, in {Sao Paulo, Mexico, Tokyo, Shanghai, Cairo, Mumbai}, the maximum length is 9 (Sao Paulo).

   Detail Drift: Distribution of value count (Cardinality Detail): A measure of how often each element occurs in the dataset, calculated as: (Number of occurrences / Total elements) × 100. For example, if "Mark" appears 3 times in a 10-row dataset, its cardinality detail is 30%. On the Alerts page in the Summary tab, you can choose to view the distribution as a percentage or number of occurrences. The Show by percentage checkbox is selected by default.

5. Set the confidence level for generating an alert by dragging the lower and higher value sliders, or enter the lower and higher values in the text boxes to set the range of percentages.
6. Define the minimum change value to generate an alert for each statistic.
   The defined change in value is calculated using the current and previous profile runs. Analyze your data and determine the appropriate level of sensitivity to ensure meaningful alerts and fewer false positives.
7. Optional: Use significant alert tuning to fine-tune your alerts and reduce noise.
8. Click Save.