Observability enables tracking of data assets within an application by monitoring their health and providing notifications when anomalies are detected. This service offers insights into system behavior, allowing to proactively identify and address issues, ensuring enhanced reliability and user experience.
- Alerts: Alerts are generated whenever an anomaly is detected based on the Observers created and the rules configured in the system
- Observers: The Observer monitors your selected data assets and generates periodic alerts. This tab provides an option to create a new Observer and also lists all the existing Observers.
- Profilers: Profiler scans your data assets and identifies problems related to correctness, completeness and validity in the data.
Data Observability configuration guide
By following this checklist, you can ensure continuous monitoring and profiling of your data, maintaining its quality and reliability. Start your data observability journey today to stay ahead of potential data issues.
- Identify the datasource: Determine the datasource you want to observe and profile.
- Establish and catalog a connection: Establish and catalog a connection for the datasource.
- Create an observer or a data profile: Set up either an Observer to monitor data or a Data Profile to analyze data based on the requirements.
- Choose cataloged data assets: Select the data assets you wish to observe or profile.
- Define rules: Determine the types of rules to apply.
- Observer rules: Establish rules to detect anomalies such as freshness, volume, data drift, and schema drift.
- Profiler rules: Establish rules to gather statistics, such as default analysis rule.
- Choose a schedule: Decide on a schedule for running the Observer and Profiler.
- Identify alert recipients: Specify who should receive alerts when anomalies are detected.
- Run the observer and profiler:
- Automatic: Run the Observer automatically.
- Manual or Scheduler: Run the Profile either manually or based on a predefined schedule.
- Receive notifications and act on alerts: Get alerts via email and view details to understand issues. Then, take necessary actions to resolve any detected issues.
- View profiling results: Examine profiling details to understand your data.
Required permissions for Observability features
| Feature | Permissions | Actions |
|---|---|---|
| Observer | View | View the details of the created observer |
| Observer | Manage |
|
| Profile | View | View the profiles and the associated details |
| Manage |
|
|
| Operate |
|
Observability features by datasources
The Observability service supports selecting data assets fetched from different categories. The data assets required to create Observers can be accessed from two different categories.
- Data Integrity Suite onboarding flow: The data assets in
this category are fetched from the datasources which are added and
discovered in the application.
Agent based assets: The
agent acts as a medium to fetch the assets from the database and
these assets are available to be selected while creating Observers
of Profilers. Tip: Profiling and execution of data quality rules are triggered only when the corresponding toggles are enabled on the insights page within the respective connections.
- Data Integrity Suite APIs: The data assets and the profiles are fetched using Data Integrity Suite APIs and the statistics are generated based on the profiles fetched from the APIs. The alerts for these data assets are generated based on the Observers created in the application.
This table outlines the compatibility of various datasources with observability. Understanding this compatibility is important to optimize data observability, ensuring that relevant metrics and logs can be effectively monitored and analyzed.
| Datasource | Deployment method | Catalog Profiling | Data Observability | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Alerts | Filter datasets | Profilers | Popularity | ||||||
| Schema Drift | Data Drift | Freshness | Volume | ||||||
| Google BigQuery | Cloud | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Databricks | Cloud | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| Snowflake | Cloud | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Amazon Redshift | Cloud | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Microsoft Azure Synapse Analytics | On-premise | Yes | Yes | Yes | No | Yes | No | No | No |
| Microsoft SQL server | On-premise | Yes | Yes | Yes | Yes | Yes | No | No | No |
| Oracle | On-premise | Yes | Yes | Yes | Yes | Yes | No | No | No |
| PostgreSQL | On-premise | Yes | Yes | Yes | No | Yes | No | No | No |
| Azure SQL server | On-premise | Yes | Yes | Yes | No | Yes | No | No | No |
| Amazon S3 | On-premise | Yes | Yes | Yes | No | Yes | No | No | No |
| SAP HANA | On-premise | Yes | Yes | Yes | Yes | Yes | No | No | No |
For more information about configuring each datasource, see Supported datasources. This documentation directs you to step-by-step instructions, including necessary parameters, and connection settings for each datasource.
- Snowflake: The tables and schemas is displayed in the application for Snowflake users having accountadmin role or roles having access to the Snowflake database. The accountadmin can grant Snowflake database access to other roles. For more information about granting Snowflake database access to other roles, see Enabling Snowflake database usage for other roles.
- BigQuery: The tables and schemas is displayed in the application for BigQuery users having resourceViewer role. Ask your administrator to grant you the BigQuery Resource Viewer (roles/bigquery.resourceViewer) IAM role on your project. For more information about granting roles, see Manage access for Google BigQuery.