Configure Databricks cluster - Precisely Data Integrity Suite

When you configure a Databricks instance pool in Clusters > Pools > Create Pool, select an LTS version from 13.3 LTS to 17.3LTS in the Preloaded Databricks Runtime Version box.

1. Log in to Databricks.
2. Click Catalog.
3. Select the specific schema.
4. Go to Permissions.
5. Click Grant and assign the required permissions.
6. Click Confirm. This will grant the required permissions to the selected users.
   • Basic user entitlements: These are inherited through the users group.
     • Workspace Access: Permission to access the Databricks workspace.
     • Databricks SQL Access: Permission to use Databricks SQL features.
   • Additional user entitlements:

     • Unrestricted Cluster Creation: You must have the Allow unrestricted cluster creation entitlement to create and manage clusters necessary for running quality tasks.
   • Permissions:
     • All-Purpose Cluster: Users need the Can Restart permission on targeted clusters.
     • SQL Warehouse: Can use permission is required on targeted SQL warehouse.
     • Instance Pool: Can Attach To permission is required for users to assign job clusters to instance pools, optimizing compute resources.
   • Required permissions to access hive_metastore or unity catalog: Appropriate privileges must be granted on catalogs, schemas, or tables to enable data access. These privileges can be granted based on how users choose to manage the data.

     Catalog: USE CATALOG

     Schema:

     • USE SCHEMA

     • SELECT

     • CREATE

     • MODIFY

     Note: Modify schema and create table permissions are necessary to create or modify target table in the quality pipeline.

     Table: SELECT

     Tip: You must have SELECT permissions on source tables to read the data. For the target table, you require sufficient privileges to perform operations that include SELECT, CREATE, and modify.
   • Advanced permissions for tables with row filter and column mask policies: If quality operations are to be performed on tables using row filter and column mask policies, the workspace must have the Serverless Compute feature enabled, and you must have appropriate policy permissions in place. For more information, see Row filters and column masks.

To manage quota constraints effectively, configure instance pools with a defined Max Capacity. When the same pipeline engine is used for multiple pipelines, jobs run in batches. Using different pipeline engines for multiple pipelines has no impact. This configuration supports efficient resource utilization and controlled job runs. Benefits include:

• Efficient resource utilization: Prevents exceeding core usage limits by controlling job concurrency.

• Streamlined job run: Jobs are processed in batches, minimizing risk of quota violations.

• Create an Instance Pool: Define the maximum capacity as the number of instances allocated for the workload.

• Onboard datasources: Batch processing involves running jobs in a sequence where some jobs run concurrently while others wait in the queue. As jobs finish, the next queued jobs start processing.

Example configurations