Hi Reader,

Integration is the lifeblood of the digital ecosystem.

And Kafka has become the poster boy of digital integration in a wide variety of business scenarios (message-based integration, stream processing, data integration, application integration).

In today’s issue of Monday Muse, I want to have a look at the Integration patterns implemented by Kafka.

These are very fundamental patterns given the way Kafka works. But looking at them in a single place gives a 360-degree perspective on these design patterns.

Let’s start!

Publish-Subscribe Pattern

It is the most common pattern that we use with Kafka. This pattern involves producers publishing messages to Kafka topics, and consumers subscribing to these topics to receive and process the messages.

It allows for a one-to-many distribution of messages, enabling multiple consumers to receive the same message independently. Producers publish messages to Kafka topics without any knowledge of the subscribed consumers.

Consumers subscribe to one or more Kafka topics to receive messages. Kafka topics act as logical channels or categories where messages are organized and stored.

Kafka maintains the messages in topics for a configurable retention period, allowing consumers to consume messages at their own pace. Each message in Kafka is assigned an offset that represents its position within a partition of a topic.

Message Queue Pattern

We all know Kafka provides a Publish-Subscribe mechanism as above.

However, Kafka can work as a message queue where producers push messages onto Kafka topics, and consumers pull messages from these topics in a FIFO (First-In-First-Out) manner.

It enables asynchronous, decoupled communication between producers and consumers. Kafka acts as a distributed, scalable message queue to facilitate asynchronous communication between producers and consumers.

As above, in publish-subscribe mode, the messages are persisted in a topic and the consumers can subscribe to one or more topics and consume all the messages present in that topic.

But in the queueing mode, messages are saved in a queue fashion. This allows messages in the queue to be pulled by consumers, but only one consumer can consume each message at a time. As soon as a consumer reads a message, it gets removed from the Apache Kafka Queue.

A Kafka queue helps in segregating the work so that each consumer receives a unique set of data to process. As a result, there is no overlap, allowing the burden to be divided and horizontally scalable.

Event Sourcing Pattern

Event sourcing is a pattern where the state of an application is determined by a sequence of events.

In Event Sourcing, events are treated as the primary source of truth, capturing all changes to the system’s state over time.

Kafka acts as the event store, persisting events in a durable and distributed manner, ensuring fault tolerance and high availability. Kafka’s append-only, immutable log makes it an excellent choice for implementing event sourcing, as it provides durability and a complete history of events.

Kafka can be used as the event store to capture write operations (commands) and allow for efficient querying (queries) by maintaining separate read models.

Event sourcing with Kafka allows for flexible event replay, enabling the rebuilding of states at different granularities, such as replaying events for specific entities, time ranges, or even specific event types.

Event store, event publication, event replay for state reconstruction, event versioning, temporal querying, the immutability of the event log (integration with CQRS), and the flexibility of event replay - all help in the implementation of Event Sourcing.

Fan-out/Fan-in Pattern:

This pattern involves consuming messages from a Kafka topic and then distributing those messages to multiple downstream processes or microservices for further processing.

It allows for parallel processing and scalability by leveraging Kafka’s partitioning mechanism.

The Fan-out/Fan-in Pattern involves consuming messages from a Kafka topic and distributing them to multiple downstream processes or microservices for further processing.

Just like the above patterns, messages are published to a single Kafka topic by producers and consumers read messages from the Kafka topic and perform initial processing.

Each consumer can produce processed messages to one or more intermediate Kafka topics. Multiple downstream consumers can subscribe to these intermediate topics, forming a fan-out effect.

Downstream consumers independently process the messages from the intermediate topics based on their specific requirements. Each downstream consumer may apply different business logic or transformations to the messages.

The Fan-in stage occurs when the processed messages from different downstream consumers are collected and consolidated into a single stream or destination.

Choreography Pattern:

The choreography pattern involves using Kafka as an event-driven messaging backbone to facilitate communication and coordination among a number of services (consumers and producers of events - in view of Kafka).

Each service publishes events to Kafka, and other services subscribe to these events to react accordingly.

For example - we can initiate a saga (Saga Pattern) by publishing an event to the relevant Kafka topic indicating the start of the saga. Each service involved in the saga performs its local transaction and publishes an event to Kafka upon completion.

Services publish events to Kafka topics to notify other services about the progress or completion of their steps. If a failure occurs during any step, services listen for compensating events on the Kafka topics to undo the previous actions.

Stream Processing Pattern

Kafka Streams is a stream-processing library provided by Kafka. This pattern involves leveraging Kafka Streams to perform real-time processing, transformations, and analytics on streams of data flowing through Kafka topics.

Stream processing starts with the ingestion of event streams into Kafka topics, which serve as the central data source for processing.

We develop and deploy stream processing applications that consume data from Kafka topics, perform computations, and produce results in real time.

Stream processing allows for maintaining and updating state information as events are processed, enabling context-aware computations and complex data transformations.

We typically perform computations over a sliding time or event window to analyze and aggregate data within specific time intervals or based on a fixed number of events.

Stream processing applications can be scaled horizontally by running multiple instances or partitions of the application, enabling parallel processing of events for increased throughput and responsiveness.

Interactive Queries: Stream processing applications can maintain materialized views or state stores that enable interactive queries, allowing for real-time retrieval of computed results or aggregated data.

Joining and Enrichment: Stream processing enables joining data from multiple streams or sources, enriching events with additional information, and performing complex data correlations for comprehensive analysis.

Integration with External Systems: Stream processing applications can seamlessly integrate with external systems and databases, enabling continuous data synchronization, data enrichment, and downstream integration with other services.

Conclusion

In the integration within a digital enterprise, a number of patterns implemented with the help of Kafka realize the integration objectives.

From Stream processing of high-velocity data streams to fanning out data streams and joining them or in the choreography of services to coordinate the sagas - Kafka is a one-stop shop for an inventory of integration scenarios.

Each of them solves a specific integration requirement and in the process addresses the non-functional aspects in slightly different ways. In some cases, improving the scalability is the motivation (fan-out), whereas in another scenario the driver is resilience & fault tolerance (event sourcing) or speed (stream processing).

Hope, you find it useful.

That’s all this week.

Till next week…