An observer is a set of rules you define and apply to a data asset. The observer monitors your selected data assets and generates alerts based on the rules you choose. Observers monitor your data assets for anomalies and generate alerts based on rules you define. Use observers to proactively identify and resolve data quality issues.
For example, an observer might check daily whether columns have been added to or deleted from the Customer_Cleansed_Table. If a change is detected, it creates a critical alert and sends it to specified recipients.
Each observer includes a schedule, the dataset to observe, the rules to apply, alert thresholds, alert severity levels, and the recipients who receive notifications.
Go to where you can create, view, edit, or delete observers.The Observers page includes these key fields and areas:
- Search: Type the complete or partial name of an observer to find matching results.
- Name: Displays the observer name. Click the ellipsis icon to edit or delete the observer. Click the observer name to view associated alerts, assets, and profile information.
-
Scheduler: Displays the observer schedule
status.
- Enabled: The observer is scheduled. Click to view observer run history.
- Suspended: The scheduler stopped after ten consecutive failures. Click the Suspended status to investigate or re-enable the schedule.
- Unavailable: The scheduler service is unavailable and cannot provide observer run history.
-
Alerts: Displays alerts and warnings for the
observer.
- Red: Critical alert. The number indicates how many critical alerts exist.
- Yellow: Warning alert. The number indicates how many warning alerts exist.
- Green: No alert. Zero indicates no alerts exist.
- Created By: Displays the name of the user who created the observer.
- Bell icon without strike-through: You're subscribed to email notifications.
- Bell icon with strike-through: You're unsubscribed from email notifications.
- Dimmed bell icon (with or without strike-through): Email notifications are disabled for this observer, or the maximum subscription limit of 25 has been reached.
Determine observer creation prerequisites
Verify prerequisites:
- The dataset is connected and cataloged.
- The dataset is not already assigned to another observer.
- You have access permissions for the dataset, including its tables and columns.
- Table and column names do not contain single (
') or double (") quotes. - Required observer rules are identified.
- An appropriate run frequency (daily or weekly) is selected.
- Email notification preferences are configured.
- Observation activities comply with applicable data privacy requirements.
Understand observer rules
Observer rules define when alerts should be generated by identifying significant changes or anomalies in your data assets. The Data Integrity Suite supports four rule types, each designed to detect different types of data issues.
Each observer includes one or more rules that monitor your data for specific conditions. When a rule detects an anomaly, it generates an alert and notifies observers.
Available observer rules
- Freshness: Alerts are generated when the data fails to update at the expected frequency.
- Volume: Alerts are generated whenever there is a change in the number of rows in the data.
- Data drift: Alerts are generated when there is a change in data beyond a specified range.
- Schema drift: Alerts are generated whenever there is a change in a table or column.
Confidence based versus threshold based rules
For freshness, volume, and data drift rules you indicate if you want to create a confidence based rule or a threshold based rule. This distinction does not apply to schema drift rules.
- Confidence based alerts: Generate alerts based on the certainty that the change meets your criteria for an alert. For example, a confidence based freshness alert says, only generate that alert if you are 80% confident (as an example) that the table failed to update as expected.
- Threshold based alerts: You manually assign limits to trigger alerts. For example, you can specifically say that you want to generate an alert if a table fails to update every three days.
Create observers
What happens after you create an observer:
- First runs: After the first scheduled run, view profiling information from the Assets and Profiling tab.
- Monitoring: After a few runs, the Observer identifies trends and patterns. When the system detects an anomaly, it creates an alert and sends a notification. You can adjust alert criteria and notification settings to fine-tune monitoring.
- Visualization: Alerts show the reason for the change and a visual representation of value changes, helping you spot trends and patterns quickly.
- Updates: Edit Observers after initial configuration to change rules, data assets, notification settings, and scheduling.
- Historical data monitoring: The Observer continues monitoring data changes over time, providing historical context for additional insights.
Search for observers: In the search field, enter the complete or partial name of an Observer. Matching results display.
Edit an observer: Click the ellipsis for the Observer you want to edit and select Edit.
Delete an observer: Click the ellipsis for the Observer you want to delete and select Delete.
View profiling details for an observer
Access profiling details associated with an Observer to review analysis results.
- Go to .
- Select the Observer whose profiling details you want to view.
- Click .
- Review the profiling details associated with the Observer.
Manage alert notifications
Configure email notifications so you're alerted when anomalies are detected or when an Observer fails to run.
- An alert is generated for a monitored data asset.
- An Observer fails to run ten times consecutively.
Observer constraints and limitations
- Data Access: Not all users have access to observe all tables and fields. Make sure that users have the right permissions to observe the data they need.
- Data Volume: Observing large datasets can be resource-intensive and may require additional processing power or storage. It will also take more time to complete a successful run on large datasets.
- Data Privacy: Observing certain types of data, such as personal data, may require additional security measures to ensure compliance with data privacy regulations.
- Data Quality: Observing tables and fields with poor data quality can lead to inaccurate or unreliable results.
- Data Governance: Observing tables and fields may be restricted by the data governance policies of your organization.