Configure run configuration - Precisely Data Integrity Suite

A run configuration specifies your source and target datasets and the pipeline engine. You can run the pipeline immediately after creating the configuration.

1. Go to Quality > Pipelines.
2. Find the pipeline for which you want to create a new run configuration.
3. In the Name column, click the ellipsis, then click Edit.
4. Click Run.
5. Click + Create New Run Configuration to expand the settings.
6. Configure settings for the new run configuration.
   • Name: Enter a name that describes the run configuration purpose or data processing type.
   • Type: Choose one of two types:
     • Batch: Process datasets in bulk.
     • Service: Expose the pipeline as a real-time API for on-demand execution.
       Note: Service based run configurations require pipelines with exactly one input and one output.

   Batch run configuration

   Field	Description
   Pipeline Engine	Select the pipeline engine on which you want to process the data. You can review the list of supported combinations to make your selection. Warning: When choosing a pipeline engine, ensure that the connection of the pipeline engine is compatible with the source dataset. Agent execution of a data quality pipeline in batch is not possible when an Enrich transformation step is part of the pipeline. Note: When creating a new run configuration during the onboarding flow for a connection, a compatible pipeline engine is automatically selected by default.
   Inputs	Input Name: Enter a unique name to identify each input in the configuration. Source Dataset: Select the dataset from which the data will be sourced. A list of connections compatible with the pipeline engine is provided to assist with your selection. Note: If an invalid connection is selected, an error will occur. While you won’t be able to schedule or manually execute invalid run configurations, you will still be able to save them.
   Outputs	Output Name: The output name that helps identify the result of the data processing. Target Options: Choose the processing options that apply to the output data. Here are the options you can choose from: Append: Output is appended to columns in the target dataset if the pipeline output schema matches the target dataset schema. Override Truncate: Clears the dataset and writes new data to it, ensuring the schema of the pipeline output matches the target dataset schema before proceeding. Note: When the target option is set to override truncate, you have the option to edit the dataset properties while browsing through the target options. Overwrite Drop: Deletes and recreates the target dataset, writing new data without matching or verifying the schema with the pipeline output. Target Dataset: Select the target dataset where the processed data will be stored. Ensure that the target location is compatible with the output specifications and accessible by the pipeline engine. Note: When setting up an output for a Databricks pipeline, ensure that the output dataset table name is specified in lowercase letters only.

   Note: The table look step is applied only at design time when setting up a pipeline, and not during the run configuration.

   Service run configuration

   Field	Description
   Description	Provide a meaningful description that reflects the run configuration purpose.
   Deploy on	Cloud: You can deploy the pipeline as a service in the Cloud using platforms such as Databricks or Snowflake. This option is suitable when you want to use cloud based resources for data processing tasks. Agent: Alternatively, you can deploy the pipeline as a service on an Agent, which operates within your own infrastructure for on-premises data sources. To use this method, select the desired agent from the drop-down menu and deploy the pipeline.
   Service Identifier URL	By default, the Service Identifier URL is automatically populated with the pipeline name. The system checks for duplicates and prevents deployment if a matching URL already exists. After deployment, the generated Service URL becomes available for use. Cloud: Once the service is deployed, it will show the Service URL https://gateway.dqcore.cloud.precisely.com/v1/api/services and additional actions become available, like preview and copy service. Agent: Once the service is deployed, it will show the Service URL https://<hostnameOfAgent>/v1/api/services/<ServiceUrl> and additional actions become available, like preview and copy service. The Swagger page displays all deployed services, and you can test or execute the selected service directly by clicking the POST button. For additional information about token generation, please refer to the following resource Generate token and API Key and Secret security.
   Preview Service	It shows only the service for which you clicked the Preview Service button. Open the service details in the Swagger page, showing the available endpoints. Note: If a pipeline service is deployed on an Agent, the user will not see the Try It Out option on the Swagger page. This means the user cannot run their service through the DIS UI and will need to use either SSH or Postman from their system instead. The Swagger UI is unable to run services because the DIS content security policy prevents the UI from accessing unknown external sites. You can preview and run individual services within a pipeline. The pipeline Preview Service displays only the specific service for which the Preview Service button is clicked. If your pipeline includes several services, choose Preview Service from pipeline design. The preview page will show only the selected service and its settings. You can also run the service directly from this page for validation purposes. This helps you quickly test and debug individual services while designing your pipeline.
   Copy Service URL	Copies the generated Service URL so you can easily access it or open it in the Swagger page.

   Note: With the service run configuration, you can now access multiple inputs within a single pipeline. However, the Pipeline Preview Service will only display the specific service for which the Preview Service button was selected. For additional information on how to add an input to a pipeline, please refer to Configure quality pipeline.
7. Click Create to save the settings for the new run configuration, or click Create and Run Pipeline to save the settings and immediately run the pipeline.
8. Alternatively, you can select Create or Create and Deploy to create a new service. Once it is successfully created, a Service tag will be displayed at the top of your pipeline canvas, indicating its availability.
9. Once batch or service run configurations are created successfully, you can find them listed under Run or Deploy.

10. You can now directly click the Run or Deploy button to execute any batch configuration or deploy any service.