Create run configuration

A run configuration defines source and target datasets as well as the pipeline engine that runs a pipeline. You can choose to immediately run the pipeline when you create a run configuration.

1. On the main navigation menu, click Quality > Pipelines.
2. Find the pipeline for which your want to create a new run configuration.
3. In the Name column, click the ellipsis!, then click Edit.
4. Click Run.
5. Click + Create New Run Configuration. This expands the New Run Configuration settings.
6. Configure settings for the new run configuration.
   • Name: Provide a meaningful name to the run configuration that reflects the purpose or the type of data processing it will perform.
   • Type: There are two types of run configurations that can be configured:
     • Batch: The batch type is ideal for processing datasets in bulk.
     • Service: The service type exposes the pipeline as a real-time API for on-demand execution.
       Note: When creating a service-based run configuration, only pipelines with one input and one output can be deployed as a service.

   Batch run configuration

   Field	Description
   Pipeline Engine	Select the pipeline engine on which you want to process the data. You can review the list of supported combinations to make your selection. Warning: When choosing a pipeline engine, ensure that the connection of the pipeline engine is compatible with the source dataset. Note: When creating a new run configuration during the on-boarding flow for a connection, a compatible pipeline engine is automatically selected by default.
   Inputs	Input Name: A unique name for each input to identify it in the configuration. Source Dataset: Select the dataset from which the data will be sourced. A list of connections compatible with the pipeline engine is provided to assist with your selection. Note: If an invalid connection is selected, an error will occur. While you won’t be able to schedule or manually execute invalid run configurations, you will still be able to save them.
   Outputs	Output Name: Output name that will help in identifying the result of the data processing. Target Options: Choose the processing options that apply to the output data. Here are the options you can choose from: Append: Output is appended to columns in the target dataset if the pipeline output schema matches the target dataset schema. Override Truncate: Clears the dataset and writes new data to it, ensuring the schema of the pipeline output matches the target dataset schema before proceeding. Overwrite Drop: Deletes and recreates the target dataset, writing new data without matching or verifying the schema with the pipeline output. Target Dataset: Select the target dataset where the processed data will be stored. Ensure that the target location is compatible with the output specifications and accessible by the pipeline engine. Note: When setting up an output for a Databricks pipeline, ensure that the output dataset table name is specified in lowercase letters only.

   Service run configuration

   Limited Availability: ^ This feature is currently available only in select workspaces and might be subject to change before general availability.

   Field	Description
   Description	Provide a meaningful description for the run configuration that reflects its purpose.
   Service Identifier URL	By default, the Service Identifier URL is automatically populated with the pipeline name. The system checks for duplicates and prevents deployment if a matching URL already exists. After deployment, the generated Service URL becomes available for use.
   Preview and Copy URL	Once the service is deployed it will show the Service URL https://pipeline-realtime-dev.dqcore.cloud.precisely.services/v1/api/service/ serviceName and additional actions become available: Preview Service: Open the service details in the Swagger page, showing the available endpoints. Copy URL: Copies the generated Service URL so you can easily access it or open it in the Swagger page. The Swagger page displays all deployed services, and you can test or execute the selected service directly by clicking the POST button.

7. Click Create to save the settings for the new run configuration, or click Create and Run Pipeline to save the settings and immediately run the pipeline.
8. Alternatively, you can select Create or Create and Deploy to create a new service. Once it is successfully created, a Service tag will be displayed at the top of your pipeline canvas, indicating its availability.
9. Once batch or service run configurations are created successfully, you can find them listed under Run or Deploy.
10. You can now directly click the Run or Deploy button to execute any batch configuration or deploy any service.