The Spark connector is a component of the Data Integrity Suite agent. It enables the agent to interact with Spark based services for efficient data processing and integration across environments.
The Spark connector:
- Provides compatibility with services that use Apache Spark
- Manages access and permission levels required by those services
- Acts as a bridge between the Agent and Spark workloads
Note:
- When using JSON array files with multiLine=true, Spark processes the entire file as one object. This approach can cause executor heap out-of-memory issues when working with large files. For smaller files, JSON remains a valid option if the file size does not exceed your executor memory limits. However, for larger datasets, JSONL (one JSON object per line) is the recommended format. JSONL enables proper partitioning and parallel processing across executors, which helps you avoid memory bottlenecks. Spark writes output in JSONL format regardless of whether your input is JSON or JSONL.
- For optimal Spark performance, ensure that your input Parquet files are properly structured with an appropriate number of row groups. Spark processes Parquet files by splitting them at row group boundaries. If your files have too few row groups, for example, a single row group, this can result in data skew and underutilization of your executor cores. To achieve efficient parallel processing and even data distribution, align the number of row groups in your Parquet files with the number of executor cores configured in your pipeline engine.
Spark connector update criteria
Manually update the Spark connector if:
- The current version lacks permissions required by a service
- A new service requires a higher connector version than currently installed
- You are experiencing compatibility issues with Spark based integrations
Warning: Ensure that no pipelines are running on the Agent during the update. If pipelines are running, stop the running pipeline or let it complete.
To manually update a specific Spark connector version, use the following commands:
- Update the CLI by running this command:
./discli updateCli -
The operator version updates automatically. If an issue arises, or if you must use a specific operator version, update it manually by running this command.
./discli agent applyOperatorVersion --version <version> -
Update the Spark connector version:
./discli agent update -s spark-connector --version 2.1.1