The Quality service in Data Integrity Suite helps you validate, analyze, and correct issues in uploaded data during integration. It helps ensure data from multiple sources remains reliable, consistent, and accurate.
This service provides a guided visual design workflow that helps users maintain data quality for downstream operations and analytics. You can deploy it in cloud-native environments or integrate it with existing data infrastructures to fit operational needs.
- Validation and enrichment: This feature validates business data against predefined rules and standards and enriches it with contextual information. It helps users understand the “who,” “where,” and “why” of business operations. The process keeps your data clean, comprehensive, and ready for further analysis.
- Geocoding: Geocoding adds geographical context to your data, so you can associate location based insights with business data. This additional geographic layer improves analytical depth and supports advanced geospatial analysis and decision-making.
- Cloud-native execution: The Quality service supports deployment in cloud-native environments and integration with on-premises systems. This flexibility helps you manage and maintain data infrastructure and scale data quality processes as business needs change.
Quality assets
In the Quality service, you can access and manage your pipelines and jobs from the Quality section. By default, information appears in card and list views, with assets sorted alphabetically by asset name.
You can monitor assets in detail across the following tabs:
- Pipeline: Lists pipelines and shows details such as pipeline name, number of datasets, pipeline status, and modification details.
- Job: Lists jobs and shows details such as job ID, pipeline name, run configuration name, start time, duration, user details, and job status.
- Schedules: Automates execution of pipeline jobs in Data Integrity Suite.
These insights help users monitor, manage, and optimize data quality processes, so data remains a reliable input for decisions.
Required permissions for Quality features
The following table provides an overview of actions available for
assigned permissions in the Quality service. It lists the permissions
required to view, manage, or operate quality pipelines, rules, scores,
and related quality functions in this service.
| Feature | Permissions | Actions |
|---|---|---|
| Quality Pipeline | View | View details of the pipelines created in the workspace |
| Manage |
|
|
| Operate |
|
|
| Rules and Scores | View | View the default and custom rules as well as generated quality scores |
| Manage |
|
|
| Operate | Manually trigger the rule execution |
Quality configuration guide
The service includes features that help transform raw business data into actionable insights, including the "who," "where," and "why" behind operations. Users can connect to auto-cataloged datasources, create or import sample data, and design processes in the cloud. Real-time data updates during design help users implement rules across diverse environments for streamlined execution.
- Select the datasource type: Choose the datasource that your quality service will target and where essential data is stored.
- Establish and catalog a connection: After setting up the data datasource, establish and catalog a connection so the Quality service can access it as required. This step is crucial for maintaining a reliable link to your data assets.
- Create quality pipeline: Design and develop a quality pipeline that orchestrates data flow through quality checks and transformation processes.
- Configure transformation steps: Implement specific transformation rules and operations that clean, standardize, and transform incoming data. This can include removing duplicates, correcting erroneous entries, and converting data formats.
- Apply run configuration and validate pipeline: Run the quality pipeline using a run configuration that specifies parameters such as batch size, execution schedule, and error-handling strategies. This configuration helps the pipeline run efficiently and produce reliable, consistent outputs.
- Manage quality jobs: Supervise quality jobs to ensure performance and results, including job execution, resource allocation, and troubleshooting issues during quality processes.
Use cases for quality
The following use cases show where data cleaning, matching, and consolidation are essential for operational success. Data cleaning processes such as parsing, normalization, and standardization, when combined with data matching and consolidation, help ensure data accuracy, consistency, and usability. The following business use cases show where these processes support operational efficiency and strategic decision-making:
| Use Case | Parameters | |||
|---|---|---|---|---|
| Parsing | Normalization | Standardization | Matching and Consolidation | |
| Customer Data Integration | Pull customer information from multiple sources, such as CRM systems, email lists, social media profiles, and third-party databases, ensuring it is in a usable format. | Standardize variations in name formats, phone numbers, and addresses across different systems, ensuring uniformity and accuracy in customer profiles. | Implement standardized data formats for customer records across all departments, making customer information consistent and easier to analyze. | Identify and merge duplicate or fragmented customer records to create a single, accurate customer profile. This supports improved customer relationship management (CRM), more personalized marketing, and better customer service. |
| Product Information Management (PIM) | Gather product details from a variety of sources such as supplier catalogs, e-commerce websites, and internal databases for further processing. | Standardize product attributes like size, color, material, and price, ensuring that product data remains consistent across different platforms and departments. | Implement uniform formats for product data to simplify management and analysis and ensure accuracy in product listings and inventory. | Combine product data from various suppliers or channels, eliminating duplicates and inconsistencies. This ensures a unified product catalog that supports accurate inventory management, pricing strategies, and consistent product listings. |
| Master Data Management (MDM) | Decompose complex datasets spanning different domains (customers, products, vendors) into manageable, structured components. | Apply normalization techniques to ensure uniformity across systems and business units, creating reliable, consistent data. | Apply company-wide data standards to ensure data uniformity across departments, leading to a consistent view of critical business entities. | Identify and merge duplicate or fragmented records across different systems, creating a comprehensive master record. This is essential for accurate reporting, informed decision-making, and regulatory compliance. |
| Financial Data Consolidation | Automatically extract financial data from various reports, invoices, transaction logs, and ledgers for more streamlined reporting. | Ensure consistency in financial data, such as currency formats, taxonomies, and accounting methods across systems, improving accuracy and transparency. | Standardize financial reporting formats to align with industry regulations and internal governance. | Consolidate financial data from multiple sources, remove duplicates, and reconcile discrepancies, ensuring accurate financial reporting, auditing, and forecasting. |
| Supply Chain Data Management | Automatically pull relevant data from sources like shipping logs, invoices, and inventory records, streamlining the tracking process. | Ensure data consistency across different suppliers, logistics providers, and platforms, simplifying supply chain tracking. | Standardize data formats across the entire supply chain, improving tracking, visibility, and overall management. | Merge and consolidate data from various supply chain partners, eliminating duplicates and resolving inconsistencies. This enhances decision-making and operational efficiency by providing a clear, accurate view of the supply chain. |
| Marketing Data Integration | Pull data from various marketing platforms (e.g., email campaigns, social media analytics, advertising tools) to create a unified dataset. | Normalize campaign metrics, customer engagement data, and ROI figures across different marketing channels to ensure consistency in reporting. | Implement standardized data formats to facilitate easier analysis, reporting, and performance tracking. | Merge marketing data from various sources to create a comprehensive view of campaign performance and customer interactions. This supports marketing strategy optimization, ROI analysis, and customer personalization. |
| Healthcare Data Management | Automatically pull patient data, treatment records, lab results, and medical histories from various healthcare systems. | Ensure uniformity in medical data, including units of measurement, drug names, and test results, across various healthcare systems. | Standardize patient data formats for easier sharing between healthcare providers, improving data accessibility and analysis. | Combine patient records from different healthcare providers, eliminating duplicates and discrepancies. This comprehensive patient profile supports more accurate diagnoses, treatment plans, and improved patient outcomes. |
| Fraud Detection and Risk Management | Automatically gather data from bank statements, transaction logs, customer accounts, and other sources to support risk analysis. | Harmonize transaction data across different platforms and systems, ensuring accuracy and comparability. | Implement standardized formats for transaction data, making it easier to identify anomalies and fraudulent patterns. | Combine and compare data from various sources to detect inconsistencies, duplicates, or suspicious patterns indicative of fraud. This improves risk mitigation and enhances fraud detection capabilities. |
| Human Resources Management | Gather employee data from sources such as resumes, HR systems, and performance reviews, ensuring it is well-structured for analysis. | Normalize job titles, salary bands, and performance metrics across different HR systems, creating consistent employee records. | Apply standardized formats for employee records to facilitate easier management, analysis, and reporting. | Merge duplicate or outdated employee records into a single, accurate database. This supports more effective workforce planning, talent management, and ensures compliance with employment regulations. |
| Real-Time Data Integration for IoT | Pull data from a variety of IoT devices and sensors in real-time, such as machinery sensors or environmental monitors. | Ensure consistency in data formats across different IoT devices, making it easier to analyze and act on the data. | Standardize the format of IoT data to ensure uniformity across devices and platforms, enhancing usability. | Combine data from multiple IoT devices, creating a unified dataset for real-time analysis and decision-making. This is critical in industries like manufacturing, where IoT data drives automation, efficiency improvements, and predictive maintenance. |
Implementing data cleaning, matching, and consolidation processes across business functions is vital for maintaining high-quality data. These practices enhance decision-making, optimize operations, and support strategic objectives in a data-driven environment.