Message Brokers & Apache Kafka

Owned by Mike Deskis
Last updated: Mar 10, 2022
2 min read

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. It was initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

Why use Kafka?

Apache Kafka is a scalable and fault-tolerant messaging system, it will enable enterprises to deploy distributed real-time applications using an event-driven architecture. Kafka can deliver events with fast absorption rates and provides persistence and guarantees sequential ordering. The actual adoption of a Kafka architecture as part of your solution will depend on your specific use case.

Here are some of the key concepts about Apache Kafka:

  • Kafka organizes messages into topics.

  • The process that is performing all of the work in Kafka is called the broker. A message producer will push data into a topic hosted on a broker and then the consumer will pull the messages from a topic via the broker.

  • Kafka topics can be divided into partitions. A partition will allow for the parallelizing of a topic spanning multiple brokers and thereby increase message absorption and throughput.

  • A broker can manage multiple partitions, but at any given time, only one of these partitions can act as the leader of a topic. The role of a leader is to be responsible for updating any replicas with new data.

  • A broker is responsible for storing messages in a persistent space. Each of the messages is stored with unique offsets and these messages in Kafka do not have a unique ID.