The Spark connector is a key component of the Data Integrity Suite Agent. It enables the Agent to interact seamlessly with Spark-based services, facilitating efficient data processing and integration across different environments. Specifically, the Spark connector:
- Ensures compatibility with services that rely on Apache Spark
- Manages access and permission levels required by those services
- Acts as a bridge between the Agent and Spark workloads
Because the Spark connector is embedded within the Agent, having the correct version is crucial for ensuring the Agent functions properly with other services that depend on Spark. If the current Spark connector version does not provide the required permissions or is incompatible with a specific service, you must manually update it to a supported version.
When to Update the Spark Connector
- The current version lacks permissions required by a service
- A new service requires a higher connector version than currently installed
- You're experiencing compatibility issues with Spark-based integrations
To manually update a specific Spark connector version, use the following commands:
- Update the CLI by executing this command:
./discli updateCli -
The operator version updates automatically. However, if an issue arises or you need to use a specific operator version, you can update it manually by running this command:
./discli agent applyOperatorVersion <operator_version_number>Note: You must migrate your agent since the new migrate agent feature enables a secure transition from an existing agent to a newer version that enforces encrypted communication. For more information on how to migrate your agent, refer to: Migrate Agent -
Update the spark-connector version:
./discli agent update spark-connector 0.0.49