Configure a Google Dataproc pipeline engine - Precisely Data Integrity Suite

Data Integrity Suite
Product
Spatial_Analytics
Data_Integration
Data_Enrichment
Data_Governance
Precisely_Data_Integrity_Suite
geo_addressing_1
Data_Observability
Data_Quality
dis_core_foundation
Services
Spatial Analytics
Data Integration
Data Enrichment
Data Governance
Geo Addressing
Data Observability
Data Quality
Core Foundation
ft:title
Data Integrity Suite
ft:locale
en-US
PublicationType
pt_product_guide
copyrightfirst
2000
copyrightlast
2026

Configure a Google Dataproc pipeline engine to run quality pipelines on Google Cloud Platform. You can customize worker nodes, autoscaling policies, and Spark properties.

When you configure a Google Dataproc pipeline engine, you specify the cloud storage bucket, worker node configuration, and optional advanced settings like autoscaling and service accounts.

  1. In the Pipeline engine name field, enter a meaningful name for the pipeline engine.
  2. Verify that Type is set to Google Dataproc.
  3. In the Connection dropdown, select the Google Cloud connection for the Dataproc cluster you want to use.
  4. In the Google cloud storage bucket field, specify the Cloud bucket that the engine uses to store ephemeral cluster and job data.
    The Dataproc connection and Dataproc cluster's service account must have read and write access to this bucket. Reference data are cached in this bucket for some operations such as Verify Address, and execution time becomes shorter after the first-time execution.
  5. In the Worker node count field, specify the number of worker nodes for processing pipelines.
  6. In the Worker node type field, enter the machine type for the worker nodes.
    For more information about supported machine types, see Supported machine types in the Google Cloud documentation.
  7. Expand the Advanced Settings section to configure optional settings:
    1. Auto-scaling Policy: Specify the name of the autoscaling policy that you want to use for automating cluster resource management.
      For more information, see Enable autoscaling in the Google Cloud documentation.
    2. Service account: Specify the service account used by services and applications running on a Compute Engine virtual machine (VM) instance to interact with other Google Cloud APIs.
      If you specify a service account, the Dataproc cluster uses the custom service account for Dataproc data handling operations instead of the default VM service account. For more information, see Service accounts in the Google Cloud documentation. The service account must have the required data access permissions for the input and output datasets used in the pipelines.
    3. Spark properties: Click Add Property to add key-value pairs that determine compute, memory, and disk resources for Dataproc batch workloads.
      For example, spark.driver.cores with a value of 8. For more information, see Spark Properties in the Dataproc Serverless documentation.
  8. Click Create or Save.
    The Google Dataproc pipeline engine is configured and ready to use in run configurations.
Related information