Configure a Databricks pipeline engine - Precisely Data Integrity Suite

Data Integrity Suite
Product
Spatial_Analytics
Data_Integration
Data_Enrichment
Data_Governance
Precisely_Data_Integrity_Suite
geo_addressing_1
Data_Observability
Data_Quality
dis_core_foundation
Services
Spatial Analytics
Data Integration
Data Enrichment
Data Governance
Geo Addressing
Data Observability
Data Quality
Core Foundation
ft:title
Data Integrity Suite
ft:locale
en-US
PublicationType
pt_product_guide
copyrightfirst
2000
copyrightlast
2026

Configure a Databricks pipeline engine to run quality pipelines on your Databricks environment. You can customize cluster settings, Spark properties, and optional enrichment catalogs.

When you configure a Databricks pipeline engine, you specify the connection, instance pool, and file storage location. You can also configure cluster settings and Spark properties to optimize performance for your workloads.

  1. In the Pipeline engine name field, enter a meaningful name for the pipeline engine.
  2. Verify that Type is set to Databricks.
  3. In the Connection dropdown, select the Databricks connection for the job processing instance you want to use.
  4. In the Job processing instance pool dropdown, select the Databricks instance pool where jobs will run.
  5. In the Fileshare path field, enter the Databricks File System (DBFS) path to a folder for storing Data Quality artifacts such as logs.
    Files are downloaded during the initial execution of a pipeline engine. A file is not downloaded if it already exists in the fileshare location. You can use DBFS mounts to object storage (for example, Amazon S3) for the processing and staging location. Use the format dbfs:/mnt/<mount-name>.
  6. Optional: In the Enrich datasets catalog field, enter the name of the data share catalog in the Databricks environment.
    Use the exact data share catalog name to access these datasets while running a pipeline with an Enrich step. This provides improved flexibility and customization for saving the data share under one name.
  7. Expand the Cluster Configuration section and select one of the following options:
    1. Auto-Scale: Specify the minimum and maximum worker nodes. Databricks automatically adds or removes worker nodes to match the current demand for computing resources.
    2. Single Node Cluster: All tasks and computations are performed on a single machine. Use this option for lightweight workloads where high performance is not a primary concern.
  8. Expand the Advanced Settings section to configure optional settings:
    1. Cluster logs path: Set the location of cluster logs to help troubleshoot unexpected issues. You can choose a different path or keep it the same as the fileshare path.
    2. Spark properties: Click Add Property to add key-value pairs that determine compute, memory, and disk resources for Databricks batch workloads.
      For example, config("spark.executor.memory", "4g") sets the executor memory to 4GB. For more information, see Spark configuration in the Databricks documentation.
  9. Click Create or Save.
    The Databricks pipeline engine is configured and ready to use in run configurations.

Suggested Spark properties to reduce disk usage

Consider implementing the following Spark properties to help minimize disk requirements during Databricks profile executions:

  • spark.shuffle.compress: Set to true
  • spark.shuffle.spill.compress: Set to true
  • spark.io.compression.codec: Use zstd
  • spark.io.compression.zstd.level: Set to 3

Enabling these properties ensures that data written to disk during Spark jobs is compressed, effectively reducing overall disk usage.

Set up the Enrich datasets catalog for Databricks

To use the optional Enrich datasets catalog feature, follow these steps:

Note: To subscribe to data or create data shares in the Precisely workspace, contact your Precisely support representative. Depending on the subscribed platform, customers can send an email to the Databricks Partnership or Snowflake Partnership to provision subscribed data. For more information about viewing data shares in the Databricks environment, see Read data shared using Databricks-to-Databricks Delta Sharing in the Databricks documentation.
  1. Set up data share: Ensure you have set up a data share that contains the datasets you intend to use for data enrichment. This share should include all the relevant datasets required for the Enrich step.
  2. Create catalog from a data share: For first-time users, go to Data > Delta Sharing > Shared with me and locate the data share containing the enrich datasets. Click Create catalog associated with the data share. This creates a catalog that includes the share within your workspace, making it accessible for future enrichment steps.
  3. Name your data share: While creating a catalog, provide a name for the catalog. This name helps you identify the specific dataset collection associated with the Enrich step.
  4. Access the catalog: Once the data share catalog is created, you can access the datasets within your workspace's Data section. The datasets are organized under the catalog name you provided.
  5. Grant permissions to users for the datasets: After the dataset is available in the Data section, authorize access to these datasets for specific users:
    1. Select the dataset that you want to provide permission for.
    2. On the Permissions tab, select Grant.
    3. In the Principals field, type and select the name of the user you want to provide permissions.
    4. Select the checkboxes SELECT, USE CATALOG, and USE SCHEMA, then click Grant.
      Note: You must grant these permissions to the same user or role that is used in the connection for cataloging and creating datasets.
Related information