Set up agents and observers to generate freshness, volume, data drift, and schema drift alerts for your data assets.
The Data Integrity Suite generates freshness,
volume, data drift, and schema drift alerts for data assets fetched
by agents. Alerts help you identify anomalies and take corrective action.
Note:
- Schema drift alerts follow schedules based on alert type. Table deleted alerts use the Discovery schedule. Column deleted, column added, and column datatype change alerts use the Observer schedule.
- An Observer is required to generate alerts. The schedule depends on the alert type.
- Agent-fetched datasets display a configuration icon for easy identification.
- Agent based datasets can create an Observer or Profiler if the pipeline engine for the connection exists in your workspace.
- The filter datasets option is available only for assets discovered through the onboarding flow.
Generate volume, data drift, and schema drift alerts
Detect anomalies in your data by
monitoring volume changes, unexpected data transformations, and
structural schema modifications. Volume alerts notify you when
record counts deviate from baseline patterns. Data drift alerts
flag when values shift outside expected distributions. Schema
drift alerts warn you when table structures change unexpectedly.
Use this workflow when you need comprehensive data quality
monitoring across multiple dimensions.
- Install the Agent and verify that it is healthy.
- Establish a new connection and start the onboarding flow.
- Enable the profile datasets option and schedule discovery. This initiates Cataloging and Profiling for selected data assets at the scheduled time.
- Create an Observer with the required assets and
schedule it to run. Tip: Schedule the Observer shortly after onboarding completes. This ensures datasets are profiled, cataloged, and available for selection.
- Monitor alerts from the Observer run and take corrective actions to resolve anomalies.
Generate freshness alerts
Monitor how current your data is by tracking when datasets
were last updated. Freshness alerts notify you when data
ingestion falls behind schedule or stops entirely, helping you
catch stale data before it impacts downstream processes. Use
this workflow when data recency is critical to your operations.
- Install the Agent and verify that it is healthy.
- Establish a new connection.
- Verify metadata is available in the Catalog section.
- Create a new Observer by selecting required assets. Disable the scheduler.
- Enable the profile datasets toggle on the insights page of the connection and start the onboarding flow.
- Enable the Observer scheduler.
- The Observer generates alerts upon successful completion.