Create run configuration

A run configuration defines source and target datasets as well as the pipeline engine that runs a pipeline. You can choose to immediately run the pipeline when you create a run configuration.

1. On the main navigation menu, click Quality > Pipelines.
2. Find the pipeline for which you want to create a new run configuration.
3. In the Name column, click the ellipsis, then click Edit.
4. Click Run.
5. Click + Create New Run Configuration. This expands the New Run Configuration settings.
6. Configure settings for the new run configuration.
   • Name: Provide a meaningful name for the run configuration that reflects the purpose or the type of data processing it will perform.
   • Type: There are two types of run configurations that can be configured:
     • Batch: The batch type is ideal for processing datasets in bulk.
     • Service: The service type exposes the pipeline as a real-time API for on-demand execution.
       Note: When creating a service-based run configuration, only pipelines with one input and one output can be deployed as a service.

   Batch run configuration

   Field	Description
   Pipeline Engine	Select the pipeline engine on which you want to process the data. You can review the list of supported combinations to make your selection. Warning: When choosing a pipeline engine, ensure that the connection of the pipeline engine is compatible with the source dataset. Agent execution of a data quality pipeline in batch is not possible when an Enrich transformation step is part of the pipeline. Note: When creating a new run configuration during the onboarding flow for a connection, a compatible pipeline engine is automatically selected by default.
   Inputs	Input Name: A unique name for each input to identify it in the configuration. Source Dataset: Select the dataset from which the data will be sourced. A list of connections compatible with the pipeline engine is provided to assist with your selection. Note: If an invalid connection is selected, an error will occur. While you won’t be able to schedule or manually execute invalid run configurations, you will still be able to save them.
   Outputs	Output Name: The output name that will help in identifying the result of the data processing. Target Options: Choose the processing options that apply to the output data. Here are the options you can choose from: Append: Output is appended to columns in the target dataset if the pipeline output schema matches the target dataset schema. Override Truncate: Clears the dataset and writes new data to it, ensuring the schema of the pipeline output matches the target dataset schema before proceeding. Overwrite Drop: Deletes and recreates the target dataset, writing new data without matching or verifying the schema with the pipeline output. Target Dataset: Select the target dataset where the processed data will be stored. Ensure that the target location is compatible with the output specifications and accessible by the pipeline engine. Note: When setting up an output for a Databricks pipeline, ensure that the output dataset table name is specified in lowercase letters only.

   Service run configuration

   Limited Availability: ^ This feature is currently available only in select workspaces and might be subject to change before general availability.

   Field	Description
   Description	Provide a meaningful description for the run configuration that reflects its purpose.
   Service Identifier URL	By default, the Service Identifier URL is automatically populated with the pipeline name. The system checks for duplicates and prevents deployment if a matching URL already exists. After deployment, the generated Service URL becomes available for use. Once the service is deployed, it will show the Service URL https://gateway.dqcore.cloud.precisely.com/v1/api/services and additional actions become available, like preview and copy service. The Swagger page displays all deployed services, and you can test or execute the selected service directly by clicking the POST button. For additional information about token generation, please refer to the following resource Generate token and API Key and Secret security.
   Preview Service	It shows only the service for which you clicked the Preview Service button. Open the service details in the Swagger page, showing the available endpoints. Note: You can now preview and run individual services within a pipeline. The pipeline Preview Service displays only the specific service for which the Preview Service button is clicked. If your pipeline includes several services, choose Preview Service from the pipeline design. The preview page will show only the selected service and its settings. You can also run the service directly from this page for validation purposes. This helps you quickly test and debug individual services while designing your pipeline.
   Copy Service URL	Copies the generated Service URL so you can easily access it or open it in the Swagger page.

   Note: With the service run configuration, you can now access multiple inputs within a single pipeline. However, the Pipeline Preview Service will only display the specific service for which the Preview Service button was selected. For additional information on how to add an input to a pipeline, please refer to Configure quality pipeline.
7. Click Create to save the settings for the new run configuration, or click Create and Run Pipeline to save the settings and immediately run the pipeline.
8. Alternatively, you can select Create or Create and Deploy to create a new service. Once it is successfully created, a Service tag will be displayed at the top of your pipeline canvas, indicating its availability.
9. Once batch or service run configurations are created successfully, you can find them listed under Run or Deploy.
10. You can now directly click the Run or Deploy button to execute any batch configuration or deploy any service.