LogoLogo
cegedim.cloudITCareAPIPrivacy
English
English
  • Documentation
  • ITCare
    • What is ITCare ?
      • Get started with ITCare
      • Demos
    • Enercare
      • Carbon footprint
    • Release notes
  • ITCare API
    • Overview
    • Authentication
    • Errors
    • Pagination
    • API Reference
      • Quick start
      • Analytics
        • Matomo
      • Changes
        • Changes
      • Compute
        • Application Servers
        • Backup Policies
        • Containers
        • Environments
        • Instances
        • Platform
        • Resource Filters
        • Resource Types
        • Resources
        • Services
        • Statuses
        • Tag Keys
        • Tag Values
        • Types
      • Databases
        • Databases
        • MariaDB
        • OpenSearch
        • PostgreSQL
        • Redis
        • SQL Server
      • Hardwares
        • Hardwares
      • Messaging
        • Apache Kafka
        • Message Brokers
        • RabbitMQ
      • Networking
        • Domains
        • Load Balancers
        • Network Clusters
        • Networks
      • Operations
        • Actions
        • Operations
      • Storage
        • Glusterfs
        • Overdrive
      • Topology
        • Topology
  • Services
    • Products
    • Support policy
    • Patch policy
    • RACI
  • Analytics
    • Matomo
      • Matomo - Features
      • Matomo - Get started
  • Compute
    • Virtual instances
      • Virtual instances - Features
        • Linux - Hardening
      • Virtual instances - Get started
    • Containers (K8s)
      • K8s - Features
        • Hardening
        • Persistent Storage
      • K8s - Get started
        • High Availability
  • Databases
    • MariaDB
      • MariaDB - Features
      • MariaDB - Get started
    • OpenSearch
      • OpenSearch - Features
        • v2 - Breaking changes
      • OpenSearch - Get started
    • PostgreSQL
      • PostgreSQL - Features
      • PostgreSQL - Get started
      • PostgreSQL - Upgrade
    • Redis
      • Redis - Features
      • Redis - Get started
      • Redis - Upgrade
    • SQL Server
      • SQL Server - Features
      • SQL Server - Get started
  • Messaging
    • Apache Kafka
      • Apache Kafka - Features
      • Apache Kafka - Get started
      • Apache Kafka - Upgrade
    • RabbitMQ
      • RabbitMQ - Features
      • RabbitMQ - Get started
    • SMS
      • SMS - Get started
  • Monitoring
    • ExtraHop
  • Security
    • Advanced Vulnerability Assessment
    • Bot Defense
      • Bot Defense - Features
    • Data Masking
      • Data Masking - Get started
    • Phishing Campaign
  • Storage
    • GlusterFS
      • GlusterFS - Features
      • GlusterFS - Get started
    • Object Storage
      • Object Storage - Features
        • S3 API compatibility
        • Limitation and Best Practices
        • Presigned URL
        • Bucket Policies
        • Bucket Lifecycle
        • Object Lock
      • Object Storage - Get started
        • Manage Object Users
        • Manage versioning in Bucket
        • Manage Bucket access
    • OverDrive
      • OverDrive - Features
      • OverDrive - Get started
Powered by GitBook
On this page
  • Resiliency
  • High level diagram
  • Security
  • Default configuration
  • Custom configuration
  • Self-service actions
Export as PDF
  1. Messaging
  2. Apache Kafka

Apache Kafka - Features

PreviousApache KafkaNextApache Kafka - Get started

Last updated 27 days ago

To take full advantage of Apache Kafka as a Distributed System, cegedim.cloud ensures that brokers are dispatched on multiple Availability Zones to maximize resiliency and redundancy.

Resiliency

The default cluster sizing includes 3 brokers provisionned over 3 Availability Zones.

The default configuration also ensure that the replication factor is set to 3 for topics and the minimum in-sync replica has to be 2.

This configuration garantees that all your topics and messages are replicated on all brokers over all the Availability Zones.

Note that provisioning more than 3 brokers will complexify things.

The more brokers, the more you will replicate your data. If you don't want to replicate more than 3 times, you will have to micro manage which topics and partitions are replicated and where in order to comply with your DRP for example.

High level diagram

Security

The provided Apache Kafka cluster is secured by the following means :

  • Inter broker communications secured with SASL_SSL

  • Client to broker communications secured with SASL_SSL

  • Secure controllers with ACLs

Controllers are managed by cegedim.cloud, simplifying your administration.

To connect to your secure Kafka cluster the following will be required :

  • The corresponding certificate

  • An existing and authorized user with a password (SCRAM-SHA-256)

Please refer to Apache Kafka - Get started to connect and interact with your Kafka cluster.

Default configuration

Important Apache Kafka broker parameters kept at default:

Parameter
Default
Description

168

The number of hours to keep a log file before deleting it.

-1

The maximum size of the log before deleting it : no limits.

1073741824 (1 gibyte)

The maximum size of a single log file.

300000 (5 minutes)

The frequency in milliseconds that the log cleaner checks whether any log is eligible for deletion.

Custom configuration

Here are the kafka broker parameters that cegedim.cloud will modify during provisioning:

Parameter
Value
Default
Description

False

True

Enable auto creation of topic on the server.

3

1

Default number of log partitions per topic. More partitions allow greater parallelism for consumption, but will also result in more files across the brokers.Replication factor is equal to the number of brokers participating in the cluster. (3 to 5)

Dynamic

1

Le facteur de réplication est égal au nombre de brokers participant au cluster. (3 à 5)

2

1

The minimum number of replicas that must acknowledge a write for the write to be considered successful for a producer that sets acks to "all" (or "-1").

Dynamic

1

Equals to the number of CPUs of the virtual machine.

2

1

Number of fetcher threads used to replicate messages from a source broker. Increasing this can increase the degree of I/O parallelism in the follower broker.

Self-service actions

Here is the list of actions available in self-service :

Action
Option
Description

Disk's size, availability zone

Enables to add a broker node to an existing cluster. Only available for Apache Kafka 3.6 and above. Node's name, CPU and RAM are predefined for cluster consistency

log.retention.hours
log.retention.bytes
log.segment.bytes
log.retention.check.interval.ms
auto.create.topics.enable
num.partitions
default.replication.factor
min.insync.replicas
num.recovery.threads.per.data.dir
num.replica.fetchers
Add a broker node
High level diagram