Kafka connection enables continuous replication and mainframe replication, facilitating real-time data transfer between source and target systems. To configure Kafka as a connection for replication pipelines, the following fields need to be configured:
- Connection: Specifies the name of data connection.
- Description: Allows you to describe the purpose of the data connection.
- Connection type: Select a connection from the dropdown list. When creating a new connection from a project, only connection types that can be used in that project type are present.
- Kafka vendor: Specifies the list of vendor or provider of the Kafka messaging system.
- Configure connection for: Enables to select the replication mode: Continuous replication, Mainframe replication, or both.Note: By default, the option 'Both' is selected. To configure the connection specifically for continuous replication, select the option 'Continuous replication'. To configure the connection specifically for mainframe replication, select the option 'Mainframe replication'.
- Replication engine: Specifies the replication engine to determine how data is replicated from the source to the target system.Note:
- For continuous replication, only continuous replication engine can be configured.
- For mainframe replication, only mainframe replication engine can be configured.
- When a new replication engine is added, it appears as the default option from the dropdown menu.
- Add Engine:Opens the Add runtime engine window.
- Add Broker: Adds another row to specify broker details such as Broker hostname or IP address.Note: Click (-) to remove the desired broker.
- Broker hostname or IP address: Specifies the hostname or IP address associated with a Kafka broker.
- Port number: Mention the port number to access the Kafka broker.
- Configuration:
- Security Protocol: Specifies the user security protocol for the connection. The available options in the dropdown are: Plaintext, SASL Plain, SSL, SASL SSL, and Kerberos. The fields displayed will vary depending on the selected security protocol.
- Use Schema Registry: By default, the option is checked to ensure data consistency and compatibility in Kafka for replication connection.
- Schema registry connection: Lists the schema registry connections configured with the same runtime engine as the Kafka connection configuration.
- Add Connection: Allows to add a new schema registry data connection.
- Additional configuration parameters: Allows you to specify extra settings that can customize the behavior of the Kafka client beyond the standard configuration options. These parameters enable you to fine-tune the connection to meet specific needs or requirements.