A run configuration specifies your source and target datasets and the pipeline engine. You can run the pipeline immediately after creating the configuration.
- Enrich and Table Lookup operators are out of scope for running Pipeline as a Service on Agent.
- For Amazon S3 and Azure Data Lake Storage (Gen2), these operators are not supported when these datasources are configured as source dataset.
-
Views under Default Catalog:Views from the default catalog will appear as options for the Target Dataset when creating a Run Configuration. However, selecting a view from the default catalog as the Target Dataset will result in job execution failure in Databricks.
Recommendation:Avoid selecting views from the default catalog for the Target Dataset due to this Databricks limitation.
- Views under Custom Catalog:Views from a custom catalog are not displayed as options for the Target Dataset during Run Configuration.
Edit existing run configuration
You can browse and edit existing run configurations for a pipeline. You can only edit a run configuration if your Data Integrity Suite administrator has granted you permission.
- Go to .
- Find the pipeline for which you want to browse or edit existing run configurations.
- Open the context menu next to the pipeline name, and then select Edit. Alternatively, select the pipeline Name directly.
- Select the Settings button to expand the Data Quality Pipeline Settings.
- On the Run Configurations page, select the run configuration you want to view or edit.
- Make any necessary changes to the run configuration settings.
- Select Save Changes.
Manage deployed service configurations
Deploy, edit, or remove service run configurations for your pipeline. This feature is currently available only in select workspaces and might be subject to change before general availability.
When you create a service run configuration successfully, it appears on the Run or Deploy page. You can only edit a service run configuration if your Data Integrity Suite administrator has granted you permission.
- Go to and select your pipeline.
- Select Settings to access the pipeline configuration options.
- Under Services, review the list of available services and their statuses.
- To deploy, edit, undeploy, or resync a service, select the appropriate action from the service panel.
- To delete a service run configuration, open the context menu next to the service name and select Delete.
Create a new dataset for a run configuration
Create a new dataset to use as a target location in your pipeline run configuration. This feature is currently available only in select workspaces and might be subject to change before general availability.