Profiling helps you understand data structures and schemas to manage integrations and data architecture, identify and resolve data quality issues such as duplicates and inconsistencies, and optimize storage and performance by archiving or deleting unused data. View detailed analysis of data structure, metadata, relationships, and quality within databases, data warehouses, and big data platforms.
To view profiling for a technical asset:
Profiling categories
Profiling includes these categories. Only
profileSetDate is required in the
API.
- Sample summary, Sample quality, Sample distribution
- Top values, bottom values, invalid/outliers, shapes
- Statistics
Sample summary
| Field | Description | Source |
|---|---|---|
| Effective Date | Date of the latest profiling information for the asset. | profileSetDate |
| Total Row Count | Total number of rows in the data set. | totalCount |
| Sample Row Count | Number of profiled rows and percentage of total count. | sampleCount |
| Base Type | Type of data. | type |
| Type Confidence | Confidence level as a percentage that profiling results accurately reflect the data type. Displayed with two decimal places (for example, 0.9753 displays as 97.53%). | confidence |
Sample quality
Next to each calculated percentage, there is a tooltip that displays the percentage relative to a total. This total can vary, such as the total of the sample or the total of valid entries.
| Field | Description | Source |
|---|---|---|
| Quality bar | A horizontal bar showing counts of valid (green), invalid/outliers (red), and null or blank (gray) rows. | |
| Valid | Counts and percentages of valid values based on Type or Semantic Type. Percentage is Valid Count divided by Sample Count. Distinct: Number of unique valid values.Note: During onboarding,
a maximum of 12,000 distinct values is
captured for profiling to focus on the
most relevant data
points.
|
matchCount
|
| Invalid/Outliers | Counts and percentages of invalid or outlier values. Percentage is Invalid/Outliers Count divided by Sample Count. | outlierCount |
| Null/Blank | Counts and percentages of null or blank entries. Percentage is Not Populated Count divided by Sample Count. | nullCount +
blankCount |
Sample distribution
The bar chart illustrates the distribution of samples based on their data type:
- Date/Time: Distribution across various time points.
- String: Distribution by string values.
- Number: Range distribution with standard deviation and mean.
- Boolean: True or false values.
Top values, bottom values, invalid/outliers and shapes
These categories display as bar charts showing values, counts, and percentages (calculated as value count divided by total samples). They appear only when data is available.
- Top values: Derived from
topKwith counts incardinalityDetail. - Bottom values: Derived from
bottomKwith counts incardinalityDetail. - Invalid/Outlier values: Values and counts
detailed in
outlierDetail. - Shapes: Values and counts detailed in
shapesDetail.
Statistics
Statistics availability depends on data type. For
example, boolean data displays only
Blank Count, Null Count, and Validation Regular
Expression when available.
| Label | Source |
|---|---|
| Null Count | nullCount |
| Blank Count | blankCount |
| Minimum Value | min |
| Maximum Value | max |
| Minimum Length | minLength |
| Maximum Length | maxLength |
| Mean | mean |
| Standard Deviation | standardDeviation |
| Multiline | multiline |
| Leading Whitespace | leadingWhiteSpace |
| Trailing Whitespace | trailingWhiteSpace |
| Leading Zero Count | leadingZeroCount |
| Validation Regular Expression | regExp |
View time-series chart
View recorded data values over a specified time period for the selected field, such as Type Confidence.