Advanced profiling captures comprehensive details on all aspects included in basic profiling, plus additional metrics and visualizations.
When you enable advance profiling in the insights page when establishing a new datasource, the profiling details page displays all basic profiling information plus the following additional details:
- Summary section
- Semantic type: Displays the detected semantic type. This field is not displayed if the semantic type is not detected in the data. Click the refresh icon next to the corresponding semantic type to re-run the semantic type detection and identify the semantic type. If you run semantic type detection again, you can view the timestamp of the latest execution by hovering over the semantic type. A semantic type job also appears in the assessments tab under Quality jobs to show its progress.
- Statistics section: Advanced profiling includes all basic statistics plus the following additional metrics:
- Null count: Displays the number of null values in every row of the table.
- Blank count: Displays the number of blank values in every row of the table.
- Minimum value: The minimum value is the lowest value in the dataset.
- Maximum value: The maximum value is the highest value in the dataset.
- Leading whitespace: Shows the number of values having leading white space characters in the selected sample data column.
- Trailing whitespace: Shows the number of values having trailing white space characters in the selected sample data column.
- Validation regular expression: Displays the validation to check whether the sample values are valid or not.
- Visualization charts: Advanced profiling includes several visualization charts to help you understand data distribution and patterns:
- Histogram: A frequency bar chart that displays the frequency of occurrence of each value in the sample data. Hover over a bar in the chart to view the frequency and value represented by the bar.
- Frequency analysis: A chart that displays the distribution of frequency of data values for any type of column. It shows the repetitions of the data value.
- Percentile: A value where an observation falls in a range of other observations. This chart is presented exclusively for fields that contain numerical data. For example, if a score falls in the 30th percentile, this means that 30 percent of all the scores recorded are lower. For a percentile chart, the X-axis represents the actual data, and the Y-axis divides the data into 100 parts, representing percentiles from 0 to 100. When you hover over a point, it displays the exact percentile for that data. For example, if you had a dataset of the publication years of books, and 2008 is at the 75th percentile, it means that 75% of the books were published before 2008, and only 25% were published in or after 2008.
-
Value analysis charts: Advanced profiling includes bar charts that display specific value categories. These categories function similarly and display only when data is available. Each category is represented as a bar chart that shows both the value and its count. Adjacent to the count, a percentage is shown. This percentage is calculated by dividing the value count by the total number of samples.
- Top values: Derived from topK and include the count of each in cardinalityDetail.
- Bottom values: Originate from bottomK and also include the count of each in cardinalityDetail.
- Invalid or Outlier values: Both the values and their counts are detailed in outlierDetail.
- Shapes: Both the values and their counts are detailed in shapesDetail.