Kafka server

Kafka is a distributed publish-subscribe messaging system with a robust queue that can handle a high volume of data. It supports passing messages from one endpoint to another.

Important! Red Hat Enterprise Linux 9.1 is recommended for Kafka. The Linux server should have internet access to download Kafka binaries and other dependencies during installation. Use only the Kafka and Linux versions that have been tested for IMS. Kafka for Windows Server is not currently supported for IMS.

Caution: Avoid restarting this server if possible. It supports core services for IMS.

Tools

Install PuTTY and WinSCP on Windows Server to connect to the Linux server before installing Kafka:

System Requirements

  • OS: Red Hat Enterprise Linux 9.1

  • External Disk: 2 TB for a production environment or 128 GB for a functional test environment

Prerequisites

  1. Create the user imskafka and add this user to the admin group on the target machine.

  2. Log on as user imskafka.

  3. Attach an external directory to /itron (refer to this Microsoft article).

  4. On the target machine, create the directory Itron-kafka in /tmp.

  5. Copy the Kafka installation scripts from the IMS package (in the scripts folder) or link to /tmp/Itron-kafka on the target machine.

  6. Connect to Kafka server using PuTTY and execute the following command:

    cd /tmp/Itron-kafka

  7. Execute the following command:

    dos2unix *.*

Note: The Kafka installation scripts should be copied from the IMS package at the customer site (IMS_[Version]\Scripts\Kafka).

Installing Kafka

  1. Run the following command to change the directory:

    cd /tmp/Itron-kafka

  2. Install Kafka using one of the following commands:

    • Unsecure Kafka: sudo ./kafka-install.sh

    • Secure Kafka: sudo ./kafka-install.sh -u <KAFKA_USERNAME> -p <KAFKA_PASWORD>

    ./kafka-install.sh

    Note: This installation uses Kafka 3.4.0.

Secure Kafka commands

  • Create a topic:

    ./kafka-topics.sh --create --bootstrap-server <BROKER_ENDPOINT>:<PORT_NUMBER> --replication-factor 1 --partitions <NUMBER_OF_PARTITIONS> --topic <NAME_OF_THE_TOPIC> --command-config ../config/admin.properties

  • List available topics:

    ./kafka-topics.sh --list --bootstrap-server <BROKER_ENDPOINT>:<PORT_NUMBER> --command-config ../config/admin.properties

  • View topic partition offset details:

    ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list <BROKER_ENDPOINT>:<PORT_NUMBER> --topic <NAME_OF_THE_TOPIC> --command-config ../config/admin.properties

Troubleshooting

Refer to Kafka troubleshooting.