Learn how to validate your quality pipeline, set up infrastructure, integrate quality rules, and execute the pipeline to ensure it runs successfully without errors.
Quality pipelines detect and fix duplicates, nonstandard formats, missing values, typos, misplaced characters, and inconsistent case. They produce clean, standardized data for integration and downstream use. Before running your pipeline, you must validate it to ensure all steps are configured correctly and no errors exist. Errors can be present in either the individual transformation steps or the quality pipeline itself.
Understand pipeline validation
Learn how quality validates pipeline changes and identifies errors and warnings that affect pipeline execution.
Example:
Consider this pipeline with three steps:
- The Rename Column step renames a column from "Bank" to "FinancialInstitution".
- The Search and Replace step searches and replaces content in the renamed "FinancialInstitution" column.
- The Case Field step changes field content to title case in the renamed "FinancialInstitution" column and in a "City" column.
If you delete the first step, quality validates that change. Because the second and third steps refer to the renamed column in the first step, the system presents an error and warning:
- An error shows for the Search and Replace step because there is no longer be a "FinancialInstitution" renamed column.
- A warning shows for the Case Field step because although there is no longer a "FinancialInstitution" column, the step will still be able to change case in the unaffected "City" column. If the Case Field step acted only on the deleted "FinancialInstitution" column, then it too would show an error instead of the warning.
The quality pipeline does not prevent you from making changes that create errors. You can choose to make a desired change that creates an error and later fix the problems caused by the error. Remember, however, you can execute a pipeline with a warning, but you cannot execute a pipeline with an error.
In this example, you could:
- Resolve the error by reconfiguring the Search and Replace step to search the previously renamed "Bank" column.
- Resolve the warning by removing the reference to the "FinancialInstitution" column or by reconfiguring the Case Field step to change case in the previously renamed "Bank" column.
- Review any error or warning symbols displayed on pipeline steps.
- Click a step with a symbol to view details about the validation issue.
- Resolve errors and warnings by reconfiguring affected steps or removing problematic references.
Integrate quality rules into pipeline
Add quality rules to your pipeline to define specific data quality checks that evaluate whether your data meets defined standards.
Quality rules are the foundation of quality assessment. Rules can be created using multiple rule types, including Specific Fields, Field by Condition, Specific Datasets, and SQL, allowing you to define checks at varying levels of complexity. Once created, rules can be applied to pipelines, scheduled for regular execution, and monitored in the Quality module. Rules provide visibility into quality status through run status indicators (Completed, In Progress, Failed) and can be enabled or disabled based on your quality assessment needs. Each rule is associated with a specific dimension and can target assets or datasets.
Run quality pipeline
Run your quality pipeline using a run configuration to start a quality job.
- Validate pipeline: Fix any issues flagged in the pipeline, particularly those related to invalid transformation step configurations or input dataset errors. Revisit and correct issues as needed.
- Check valid data subscription: Verify if you are using transformation steps that require data subscriptions. While you can add these steps during pipeline design, a valid subscription is necessary for successful execution. Examples include Enrich, Identify Country, and Verify & Geocode Address steps.
- Set up an agent: This step is applicable only if you intend to run the pipeline on-premises. Create an agent that is used in the run configuration settings of the pipeline.
- Create pipeline engine: The pipeline engine is the processing engine required to execute the pipeline. Create a pipeline engine using supported connections.
- Run configuration settings: Every pipeline requires run configuration settings. These settings define the target dataset where pipeline changes are applied, and the pipeline engine used for execution.
- Filter sensitive table data using row filters and
column masks: To enable filtering sensitive
table data, ensure that your workspace is configured for
serverless compute and verify that the instance pool is
set up with Databricks Runtime 15.4 LTS or a later
version.Note: This guideline applies specifically to running jobs in a Databricks environment.
- Go to and find the pipeline you want to run.
- In the Name column, click the context menu button, then click Edit.
-
Click Run and choose how to
proceed:
- To use an existing run configuration, click the run configuration that you want to use to run the pipeline.
- To create a new run configuration, click + Create New Run Configuration, configure settings for the new run configuration, then click Create to save the settings or Create and Run Pipeline to save and immediately run the pipeline.
- Alternatively, on the context menu, you can click .