Apache Kafka - Get started

Deploy a cluster

To get started, go to ITCare and search for your target global service where you'll create your new Apache Kafka cluster.

Search for your Global Service in the top search bar and click on it to display its information page.

Once in your Global Service, click on the Create Resource button, select Apache Kafka and the required version.

Fill in the form:

• Define the name of the future cluster

• Number of brokers (3+)

• Sizing

• Storage required on each broker

• Target location

• Target network

• Management options (backup, monitoring, 24/7, remote site replication)

Click Next once all fields have been completed.

In the next step, enter the password for the super user account to be supplied, then click Next.

Passwords are not saved by cegedim.cloud. Be sure to save your password!

Review the summary before submitting the form.

Provisioning can take up to 2 hours, depending on the current automation load.

Once the deployment is ready, you'll be notified by e-mail.

Start a cluster

At the top of the cluster page, click on the Manage button, then on Start and confirm.

Cluster startup starts all virtual machines attached to the cluster.

An e-mail notification will be sent when the service is activated.

Stop a cluster

At the top of the cluster page, click on the Manage button, then on Stop.

Enter an RFC number for tracking (optional). Click on Submit.

Shutting down a cluster will stop all virtual machines attached to the cluster, and monitoring will be disabled.

An e-mail notification will be sent when the cluster is shut down.

Resize nodes

At the top of the cluster page, click on the Manage button, then on Resize nodes.

Select the nodes you wish to resize and select the new size (cpu/ram).

Each node will be resized and restarted sequentially.

An e-mail notification will be sent when all nodes have been resized.

Delete a cluster

At the top of the cluster page, click on the Manage button, then on Delete. This will stop and delete all virtual machines.

Please note that this action is not recoverable!

Enter an RFC number for tracking (optional), then click Submit.

An e-mail notification will be sent when the cluster is deleted.

How to manage Apache Kafka ?

To interact with your secure cluster using Kafka scripts, you first need to download the Apache Kafka archive from the official website.

https://kafka.apache.org/downloadskafka.apache.org

Ideally, you should download the exact version corresponding to your cluster.

Once unzip and unarchived on your linux server, you will find the Kafka shell scripts under the /bin directory.

These scripts allows to :

• Produce and consume

• Manage users

• Manage topics

• Manager ACLs

• Manage item configurations

This guide will not get into the details of every scripts but help you get started with simple commands.

Authentication

To connect to a secured Kafka cluster, you need to configure a keystore and a property file.

Create keystore

Create the keystore with the provided certificate :

keytool -keystore kafka.client.truststore.jks -alias ca-cert-cluster1 -import -file ca-cert -storepass <redacted> -keypass <redacted> -noprompt

• Alias : alias of the certificate inside the keystore

• Import-file : name of the certificate file containing the provided certificate

• Storepass and keypass : password to protect your keystore, should be identical

To list the content of your keystore, use this command :

keytool -list -v -keystore kafka.client.truststore.jks

Property file

With the keystore created, now you need a property file :

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="password";
ssl.truststore.location=/path/to/kafka.client.truststore.jks
ssl.truststore.password=keystore-password

• username : the kafka super user provided to you by email

• password : the password for that user that you provided at provisioning

• ssl.truststore.location : the location to your keystore previously created

• ssl.truststore.password : the password to unlock your keystore (storepass / keypass used)

Command line

With these elements, you can now use any Kafka shell script with the following parameter :

--command-config client.properties

Manage topics

Create a topic

# Use an env variable for short commands, the bootstrap-server is available in ITCare
export BROKERS=broker1.hosting.cegedim.cloud:9094,broker2.hosting.cegedim.cloud:9094,broker3.hosting.cegedim.cloud:9094 

kafka-topics.sh --bootstrap-server $BROKERS --create --replication-factor 3 --partitions 3 --topic my-topic --command-config client.properties 
Created topic my-topic.

List topics

kafka-topics.sh --bootstrap-server $BROKERS -list --command-config client.properties my-topic

Describe a topic

kafka-topics.sh --bootstrap-server $BROKERS --describe --topic my-topic --command-config client.properties 
Topic: my-topic	      TopicId: 84yqCErzTG27J4wv44dkPQ	PartitionCount: 4	ReplicationFactor: 3	Configs: cleanup.policy=delete
Topic: my-topicc	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
Topic: my_topic		Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
Topic: my_topic		Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
Topic: my_topic		Partition: 3	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3

Delete a topic

kafka-topics.sh --bootstrap-server $BROKERS --delete --topic my-topic --command-config client.properties

Add partitions to topic

kafka-topics.sh --bootstrap-server $BROKERS --alter --topic my-topic --partitions 16 --command-config client.properties

List under-replicated partitions for all topics

kafka-topics.sh --bootstrap-server $BROKERS --describe --under-replicated-partitions --command-config client.properties

List ACLs for a topic

kafka-acls.sh --bootstrap-server $BROKERS --topic=my-topic --list --command-config client.properties

Manage users

Create a Kafka user

kafka-configs.sh --bootstrap-server $BROKERS --alter --add-config 'SCRAM-SHA-256=[password=secret]' --entity-type users --entity-name username --command-config client.properties

List Kafka users

kafka-configs.sh --bootstrap-server $BROKERS --describe --entity-type users --command-config client.properties

Delete a Kafka user

kafka-configs.sh --bootstrap-server $BROKERS --alter --delete-config 'SCRAM-SHA-256' --entity-type users --entity-name username --command-config client.properties

List all ACLs

kafka-acls.sh --bootstrap-server $BROKERS --list --command-config client.properties

List ACL for Kafka user

kafka-acls.sh --bootstrap-server $BROKERS --principal User:admin --list --command-config client.properties

Set ACL for Kafka user

kafka-acls.sh --bootstrap-server $BROKERS --add --allow-principal User:alice --producer --topic my-topic --command-config client.properties

Remove ACL for Kafka user

kafka-acls.sh --bootstrap-server $BROKERS --remove --allow-principal User:bob --consumer --topic my-topic --group my-consumer-group --command-config client.properties

Produce

Start a producer

kafka-console-producer.sh --broker-list $BROKERS --topic my-topic --producer.config client.properties 
>

Consume

Start a consumer

kafka-console-consumer.sh --bootstrap-server $BROKERS --topic my-topic --consumer.config client.properties --group consu

List all consumer groups

kafka-consumer-groups.sh --list --bootstrap-server $BROKERS --command-config client.properties

Describe consumer group

kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server $BROKERS --describe --group consu --command-config client.properties

Delete a consumer group

kafka-consumer-groups.sh --bootstrap-server $BROKERS --group my-group --group my-other-group --delete --command-config client.properties

Kcat

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8.

GitHub - edenhill/kcat: Generic command line non-JVM Apache Kafka producer and consumerGitHub

Version 1.5.0 and above must be used to support SASL_SSL authentication.

More information regarding Kafkacat is available on Confluent website :

Use kcat (formerly kafkacat) to test and debug Apache Kafka deployments | Confluent Documentationdocs.confluent.io

Kafka clients

Please refer to this documentation to create a Kafka client in any language you need :

https://docs.confluent.io/home/clients/overview.htmldocs.confluent.io

Kafka connectors

Please refer to this documentation to know more about Kafka connectors :

Kafka Connect | Confluent Documentationdocs.confluent.io

Kafka add broker node

At the top of the cluster page, click on the Manage button, then on Add broker node. Select the disk's size and the availability zone.

Enter an RFC number for tracking (optional), then click Submit.

This feature is only available for Kafka 3.6 or later. Only one node can be added at a time. Node's name, RAM and CPU are predefined for cluster consistency.

An e-mail notification will be sent when the node have been created.