In order to support event-driven architecture and its new streaming data paradigm, additional technologies are needed. One of the most popular tools for working with streaming data is Apache Kafka. It is an open‑source distributed event-streaming platform used by thousands of companies. Think about the implementation of high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Kafka allows organizations to build real-time streaming systems, and then these systems will react to the data streams to do real-time data analytics, react, aggregate, join real-time data flows, and perform complex event processing.
The popularity of Kafka is linked to the ease of the setup and its usage. This is why many large companies that handle a lot of data use Kafka. So how does Kafka work?
Example of Kafka
When a client application sends a request to a RESTful, GraphQL, SOAP, or other synchronous API, it waits for a response. When you work with asynchronous APIs, the client does not need a response, and the server does not send it. Such APIs implement the event-driven architecture.
A typical process looks like this:
A publisher (or producer, in Kafka terms) posts an event (sends a request) to a broker. This event is a piece of data to transmit and it can be many different things (ie. a JSON object containing information about a transaction, details of a newly created object in a database, and so on).
In order to separate one type of event from another, a producer sends it to one of the channels (or topics) specified within a broker.
If another application needs events related to some topic, it subscribes to the needed channel. Once a publisher posts an event to that channel, the subscriber (or consumer) receives the event and can work with it as it is needed by a business logic: