Set up Kafka as a data connection for continuous replication pipelines in the Data Integrity Suite.
The Data Integrity Suite supports the following Kafka datasources.
| Datasource | Description |
|---|---|
| Apache Kafka | Connect to a self-managed Apache Kafka cluster. |
| AWS MSK Kafka | Connect to an Amazon Managed Streaming for Apache Kafka (MSK) cluster. |
| Azure Event Hubs Kafka | Connect to Azure Event Hubs by using the Kafka endpoint. |
| Confluent Cloud Kafka | Connect to a Confluent Cloud Kafka cluster. |
| Confluent Platform Kafka | Connect to a self-managed Confluent Platform deployment. |
Connection configuration
Configure the following fields to establish your Kafka connection.
| Field | Description |
|---|---|
| Connection name | Name of the Kafka connection. |
| Description | Optional description of the connection. |
| Agent to access data | Agent used to connect to the Kafka cluster. |
| Broker hostname or IP address | Hostname or IP address of the Kafka broker. |
| Port | Port used to communicate with the Kafka broker. |
| Add broker | Adds additional broker endpoints for high availability or clustered deployments. |
Authorization
The available authorization options depend on your Kafka data source and the selected security protocol.
Authentication method
Select how credentials are stored.
| Option | Description |
|---|---|
| Regular | Stores credentials in the connection configuration. |
| Key Vault | Retrieves credentials from an external key vault. Available only for supported Kafka data sources. |
Security protocol
Select the protocol used to secure communication with the Kafka cluster.
| Protocol | Description |
|---|---|
| PLAINTEXT | No encryption or authentication. |
| SSL | Uses SSL/TLS encryption. |
| SASL_PLAINTEXT | Uses SASL authentication without SSL encryption. |
| SASL_SSL | Uses SASL authentication over SSL/TLS encryption. |
| Security Protocol | Apache Kafka | Confluent Platform | AWS MSK Kafka | Confluent Cloud Kafka | Azure Event Hubs Kafka |
|---|---|---|---|---|---|
| PLAINTEXT | Yes | Yes | Not supported | Not supported | Not supported |
| SASL_PLAINTEXT (PLAIN) | Yes | Yes | Not supported | Not supported | Not supported |
| SSL | Not currently supported | Not currently supported | Not currently supported | Not applicable | Not applicable |
| SASL_SSL (PLAIN) | Not currently supported | Not currently supported | Not applicable | Yes | Yes |
| SASL_SSL (SCRAM-SHA-512) | Not applicable | Not applicable | Yes | Not applicable | Not applicable |
SSL for Apache Kafka, Confluent Platform, and AWS MSK, and SASL_SSL (PLAIN) for Apache Kafka and Confluent Platform are not currently supported. Support for these configurations will be added in a future CDC release
Configure each Kafka data source
Apache Kafka
Apache Kafka supports the following security protocols:
-
PLAINTEXT
-
SASL_PLAINTEXT
Depending on the selected protocol, configure the following additional fields.
SSL
| Field | Description |
|---|---|
| Truststore file | Upload the truststore (.jks) file used to validate the broker certificate. |
| Truststore password | Password for the truststore, if applicable. |
| Keystore file | Upload the keystore (.jks) file for mutual TLS authentication. |
| Keystore password | Password for the keystore. |
| Keystore private key password | Password for the private key stored in the keystore. |
SASL
| Field | Description |
|---|---|
| SASL mechanism | Authentication mechanism, such as PLAIN or SCRAM. |
| Username | Kafka username. |
| Password | Kafka password. |
AWS MSK Kafka
AWS MSK Kafka supports secure authentication using SASL over SSL.
Configure the following fields.
| Field | Description |
|---|---|
| Security protocol | Select SASL_SSL or SSL, depending on your MSK cluster configuration. |
| Authentication method | Choose Regular or Key Vault. |
| SASL mechanism | Select the authentication mechanism supported by your cluster, such as SCRAM-SHA-512. |
| Username | SCRAM user name. |
| Password | SCRAM password. |
Azure Event Hubs Kafka
Azure Event Hubs exposes a Kafka-compatible endpoint.
Configure the following fields.
| Field | Description |
|---|---|
| Security protocol | Select SASL_SSL. |
| Authentication method | Choose Regular or Key Vault. |
| SASL mechanism | Select PLAIN or OAUTHBEARER, depending on your Azure Event Hubs authentication method. |
| Username | Enter $ConnectionString when using a connection string. |
| Connection string | Azure Event Hubs connection string. |
Confluent Cloud Kafka
Configure the following fields.
| Field | Description |
|---|---|
| Security protocol | Select SASL_SSL. |
| Authentication method | Choose Regular or Key Vault. |
| SASL mechanism | Select PLAIN. |
| API Key | Confluent Cloud API key. |
| API Secret | Confluent Cloud API secret. |
Confluent Platform Kafka
Confluent Platform supports the same authentication options as Apache Kafka.
Supported security protocols include:
-
PLAINTEXT
-
SASL_PLAINTEXT
Depending on the selected protocol, configure the appropriate SSL or SASL fields.
Advanced options
| Field | Description |
|---|---|
| Additional configuration parameters | Additional Kafka client properties specified as key-value pairs. These settings override the default Kafka client configuration when required. |
Replication
Enable Use for replication if the Kafka connection will be used for replication.
When enabled, configure the following fields.
| Field | Description |
|---|---|
| Replication engine | Runtime engine used for replication. |
| Add replication engine | Creates a new replication runtime engine if one does not already exist. |