Profiling is the foundational process that enables alert generation in the Data Integrity Suite. It captures data characteristics and supports the detection of anomalies and quality issues. When an observer runs successfully, it generates profiling statistics. Any changes to these profiling characteristics trigger alerts.
Profiling overview
Profiling analyzes data assets to establish baseline characteristics and collect detailed metrics that serve as reference points for detecting anomalies.
Profiling captures comprehensive information about datasets and individual fields, including data quality metrics, statistical properties, and data distributions. This profiling data is essential for:
- Establishing baseline characteristics for comparison
- Detecting data drift and anomalies
- Identifying data quality issues
- Supporting alert generation and evaluation
Accessing profiling details for an Observer:
- From the main navigation menu, click .
- Select the Observer whose profiling details you want to view.
- On the Observers page, go to .
- The page displays the profiling details associated with the Observer.
Tip: For Snowflake and Databricks datasources, profiling statistics are generated automatically when you run an Observer. However, for other datasources, you must enable the Profile datasets option on the Insights page when you set up a new datasource. This allows profiling statistics to be generated after a successful Observer run.