> For the complete documentation index, see [llms.txt](https://academy.cegedim.cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.cegedim.cloud/messaging/apache-kafka/apache-kafka-features.md).

# Apache Kafka - Features

To take full advantage of Apache Kafka as a Distributed System, **cegedim.cloud** ensures that brokers are dispatched on multiple Availability Zones to maximize resiliency and redundancy.

## Resiliency <a href="#apachekafkaarchitecture-resiliency" id="apachekafkaarchitecture-resiliency"></a>

The default cluster sizing includes 3 brokers provisionned over 3 Availability Zones.

The default configuration also ensure that the replication factor is set to 3 for topics and the minimum in-sync replica has to be 2.

This configuration garantees that all your topics and messages are replicated on all brokers over all the Availability Zones.

{% hint style="warning" %}
Note that provisioning more than 3 brokers will complexify things.

The more brokers, the more you will replicate your data. If you don't want to replicate more than 3 times, you will have to micro manage which topics and partitions are replicated and where in order to comply with your DRP for example.
{% endhint %}

## High level diagram <a href="#apachekafkaarchitecture-highleveldiagram" id="apachekafkaarchitecture-highleveldiagram"></a>

```mermaid
graph LR
    client["👤 Client"]

    subgraph vlan["Customer VLAN"]
        direction TB
        subgraph azA["Availability Zone A"]
            direction LR
            c0["Controller 0<br/>Port 9093"] <-->|internal broker coms| b0["Broker 0"]
            b0 --> j0["JMX Exporter<br/>Port 9581"]
            b0 --> t0["🔒 TLS<br/>Port 9094"]
        end
        subgraph azB["Availability Zone B"]
            direction LR
            c1["Controller 1<br/>Port 9093"] <-->|internal broker coms| b1["Broker 1"]
            b1 --> j1["JMX Exporter<br/>Port 9581"]
            b1 --> t1["🔒 TLS<br/>Port 9094"]
        end
        subgraph azC["Availability Zone C"]
            direction LR
            c2["Controller 2<br/>Port 9093"] <-->|internal broker coms| b2["Broker 2"]
            b2 --> j2["JMX Exporter<br/>Port 9581"]
            b2 --> t2["🔒 TLS<br/>Port 9094"]
        end
    end

    t0 <--> client
    t1 <--> client
    t2 <--> client

    classDef controller fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000
    classDef broker fill:#ffcdd2,stroke:#c2185b,stroke-width:2px,color:#000
    classDef port fill:#eeeeee,stroke:#666,stroke-width:1px,color:#000
    class c0,c1,c2 controller
    class b0,b1,b2 broker
    class j0,j1,j2,t0,t1,t2,client port
```

## Security <a href="#apachekafkaarchitecture-security" id="apachekafkaarchitecture-security"></a>

The provided Apache Kafka cluster is secured by the following means :

* Inter broker communications secured with SASL\_SSL
* Client to broker communications secured with SASL\_SSL
* Secure controllers with ACLs

Controllers are managed by cegedim.cloud, simplifying your administration.

To connect to your secure Kafka cluster the following will be required :

* The corresponding certificate
* An existing and authorized user with a password (SCRAM-SHA-256)

Please refer to [Apache Kafka - Get started](/messaging/apache-kafka/apache-kafka-get-started.md) to connect and interact with your Kafka cluster.

## Default configuration <a href="#apachekafkaarchitecture-defaultconfiguration" id="apachekafkaarchitecture-defaultconfiguration"></a>

Important Apache Kafka broker parameters kept at default:

<table data-full-width="true"><thead><tr><th width="301.33333333333337">Parameter</th><th width="225">Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_log.retention.hours">log.retention.hours</a></td><td>168</td><td>The number of hours to keep a log file before deleting it.</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_log.retention.bytes">log.retention.bytes</a></td><td>-1</td><td>The maximum size of the log before deleting it : no limits.</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_log.segment.bytes">log.segment.bytes</a></td><td>1073741824 (1 gibyte)</td><td>The maximum size of a single log file.</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_log.retention.check.interval.ms">log.retention.check.interval.ms</a></td><td>300000 (5 minutes)</td><td>The frequency in milliseconds that the log cleaner checks whether any log is eligible for deletion.</td></tr></tbody></table>

### Custom configuration

Here are the kafka broker parameters that cegedim.cloud will modify during provisioning:

<table data-full-width="true"><thead><tr><th width="305.5">Parameter</th><th width="103">Value</th><th width="91">Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_auto.create.topics.enable">auto.create.topics.enable</a></td><td>False</td><td>True</td><td>Enable auto creation of topic on the server.</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_num.partitions">num.partitions</a></td><td>3</td><td>1</td><td>Default number of log partitions per topic. More partitions allow greater parallelism for consumption, but will also result in more files across the brokers.Replication factor is equal to the number of brokers participating in the cluster. (3 to 5)</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_default.replication.factor">default.replication.factor</a></td><td>Dynamic</td><td>1</td><td>Le facteur de réplication est égal au nombre de brokers participant au cluster. (3 à 5)</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_min.insync.replicas">min.insync.replicas</a></td><td>2</td><td>1</td><td>The minimum number of replicas that must acknowledge a write for the write to be considered successful for a producer that sets acks to "all" (or "-1").</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_num.recovery.threads.per.data.dir">num.recovery.threads.per.data.dir</a></td><td>Dynamic</td><td>1</td><td>Equals to the number of CPUs of the virtual machine.</td></tr><tr><td><a href="https://kafka.apache.org/documentation/#brokerconfigs_num.replica.fetchers">num.replica.fetchers</a></td><td>2</td><td>1</td><td>Number of fetcher threads used to replicate messages from a source broker. Increasing this can increase the degree of I/O parallelism in the follower broker.</td></tr></tbody></table>

### Self-service actions

Here is the list of actions available in self-service :

<table data-full-width="true"><thead><tr><th width="305.5">Action</th><th width="103">Option</th><th>Description</th></tr></thead><tbody><tr><td><a href="https://academy.cegedim.cloud/messaging/apache-kafka/apache-kafka-get-started/#apachekafkahowtos-kafkaaddbrokernode">Add a broker node</a></td><td>Disk's size, availability zone</td><td>Enables to add a broker node to an existing cluster. Only available for Apache Kafka 3.6 and above. Node's name, CPU and RAM are predefined for cluster consistency</td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://academy.cegedim.cloud/messaging/apache-kafka/apache-kafka-features.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.