Profiling captures data characteristics and provides metrics that help you understand data quality, structure, and usage patterns. You can access profiling metrics at the dataset or field level to understand data quality, structure, and characteristics. Choose between basic profiling for foundational metrics or advanced profiling for comprehensive analysis with visualizations.
The Data Integrity Suite offers two profiling levels:
- Basic profiling: Captures essential metrics including
summary information, quality indicators, and foundational
statistics. This level is enabled by default when you
establish a new datasource. It provides a foundational
understanding of your dataset through key metrics and
statistics.
-
Summary
- Effective date: The date of the latest set of profiling information received for the asset.
- Sample row count: The total number of rows in the dataset that were profiled.
- Base type: The type of data in the column.
- Quality: The quality bar indicates the
overall quality and reliability of the profiled
data. It displays the following metrics:
- Valid: Displays the numeric count and percentage of values that are valid.
- Distinct: Indicates the number of unique valid values. During the onboarding flow, a maximum of 12,000 distinct values is captured for profiling. This approach helps streamline data analysis, allowing you to focus on the most relevant and meaningful data points without being overwhelmed by excessive details.
- Invalid or Outliers: Displays the sum of the total number of invalid values and outliers.
- Null or Blank: Displays the sum of the total number of null and blank values.
-
Statistics Several statistics are provided through the profiling details. The availability of specific statistical values depends partly on the data type. Click the diamond icon next to the associated metrics to view the historical values over a certain period.
- Minimum length: The minimum length of the string in the column.
- Maximum length: The maximum length of the string in the column.
- Mean: The average of the elements present in the numeric data.
- Standard deviation: A measure that provides a variation in data points from its mean value. The standard deviation is calculated as the square root of variance by determining each data point's deviation relative to the mean
-
- Advanced profiling: Includes all basic profiling
metrics plus additional details such as semantic type
detection, extended statistics, and visualization charts for
deeper analysis. Advanced profiling captures comprehensive
details on all aspects included in basic profiling, plus
additional metrics and visualizations.
-
Summary
- Semantic type: Displays the detected semantic type. This field is not displayed if the semantic type is not detected in the data. Click the refresh icon next to the corresponding semantic type to re-run the semantic type detection and identify the semantic type. If you run semantic type detection again, you can view the timestamp of the latest execution by hovering over the semantic type. A semantic type job also appears in the assessments tab under Quality jobs to show its progress.
-
Statistics:Advanced profiling includes all basic statistics plus the following additional metrics:
- Null count: Displays the number of null values in every row of the table.
- Blank count: Displays the number of blank values in every row of the table.
- Minimum value: The minimum value is the lowest value in the dataset.
- Maximum value: The maximum value is the highest value in the dataset.
- Leading whitespace: Shows the number of values having leading white space characters in the selected sample data column.
- Trailing whitespace: Shows the number of values having trailing white space characters in the selected sample data column.
- Validation regular expression: Displays the validation to check whether the sample values are valid or not.
-
Visualization charts: Advanced profiling includes several visualization charts to help you understand data distribution and patterns:
- Histogram: A frequency bar chart that displays the frequency of occurrence of each value in the sample data. Hover over a bar in the chart to view the frequency and value represented by the bar.
- Frequency analysis: A chart that displays the distribution of frequency of data values for any type of column. It shows the repetitions of the data value.
- Percentile: A value where an observation falls in a range of other observations. This chart is presented exclusively for fields that contain numerical data. For example, if a score falls in the 30th percentile, this means that 30 percent of all the scores recorded are lower. For a percentile chart, the X-axis represents the actual data, and the Y-axis divides the data into 100 parts, representing percentiles from 0 to 100. When you hover over a point, it displays the exact percentile for that data. For example, if you had a dataset of the publication years of books, and 2008 is at the 75th percentile, it means that 75% of the books were published before 2008, and only 25% were published in or after 2008.
-
Value analysis charts:Advanced profiling includes bar charts that display specific value categories. These categories function similarly and display only when data is available. Each category is represented as a bar chart that shows both the value and its count. Adjacent to the count, a percentage is shown. This percentage is calculated by dividing the value count by the total number of samples.
- Top values: Derived from topK and include the count of each in cardinalityDetail.
- Bottom values: Originate from bottomK and also include the count of each in cardinalityDetail.
- Invalid or Outlier values: Both the values and their counts are detailed in outlierDetail.
- Shapes: Both the values and their counts are detailed in shapesDetail.
-
You can access profiling metrics at two scopes:
- Dataset level: View both dataset profile metrics and column-level metrics from the dataset details page or side panel. This functionality offers a comprehensive overview of the dataset, making it ideal for those seeking to thoroughly understand its structure, usage, and quality.
- Field level: Examine metrics specific to individual columns for detailed analysis. Profiling details are available at the field level, enabling you to examine metrics specific to each column. This feature provides a detailed overview of each field and its profiling attributes, providing an in-depth understanding of data structure, usage, and quality.
To access profiling:
-
Go to Catalog and choose your access path:
- To view profiling at the dataset level, go to the Datasets tab, then click the dataset name to open the details page.
- To view profiling at the field level, go to the Fields tab, then click the field name to open the details page.
- Select the Profile tab to view profiling metrics.
- Review the profiling data displayed. The metrics shown depend on whether basic or advanced profiling is enabled for the datasource.
View dataset profiling details
Access comprehensive profiling metrics for a dataset and its fields from either the full details page or the side panel.
You can view profiling metrics in two ways. The full dataset details page provides a complete view of profiling metrics at both the dataset and field levels, ideal for thorough analysis. The side panel offers a streamlined view for quick reference while browsing multiple datasets.
View field profiling details
Access detailed profiling metrics for a specific field from either the full details page or the side panel.
You can view field profiling metrics in two ways. The full field details page provides an in-depth view of profiling metrics specific to a single column, enabling you to examine field-level characteristics, usage patterns, and data quality indicators in detail. The side panel offers a condensed, high-level overview for quick reference without navigating away from the fields list.