Multi-threaded Apache Kafka Consumer

-



Why do we need multi-thread consumer model? 

Suppose we implement a notification module which allow users to subscribe for notifications from other users, other applications.

Our module reads messages which will be written by other users, applications to a Kafka clusters. In this case, we can get all notifications of the others written to a Kafka topic and our module will create a consumer to subscribe to that topic.  

Everything seems to be fine at the beginning. However, what will happen if the number of notifications produced by other applications, users is increased fast and exceed the rate that can be processed by our module?  

All the messages/notifications that haven’t been processed by our module, are still in the Kafka topic. However, things get more danger when the number of messages is too much. Some of them will be lost when the retention policy is met (Note that Kafka retention policy can be time-based, partition size-based, key-based). And more important, when our notification module falls very far behind the income notifications/messages, it is not a true “notification” module anymore.

It’s time to think about the multi-thread consumer model.


Multi-threaded Apache Kafka consumer model

There are 2 possible models.  

  • Multiple consumers with their own threads - model 1

  • Single consumer, multiple worker processing threads - model 2


Test Multi-threaded Apache Kafka Consumer


Conclusion 

Multiple consumers with their own threads. The model is easy to implement. The total of consumers is limited the total partitions of the topic. The redundant consumers may not be used.

Single consumer, multiple worker processing threads. The model is flexible in scale out the number of processing thread. But it’s not easy to implement in-order processing on per partition. Let’s say there are 2 messages on the same partitions being processed by 2 different threads. To guarantee the order, those 2 threads must be coordinated somehow.

Comments

Post a Comment

Popular posts from this blog

Debug Java Stream in Intellij Idea

-
Image
Stream is a very powerful feature. it allows you to take full advantage of modern multi-core architectures and lets you process data in a declarative way.  Unfortunately stream API may sometimes be difficult to debug. This happens because they require you to insert additional breakpoints and thoroughly analyze each transformation inside the stream.  IntelliJ IDEA provides a solution to this by letting you visualize what is going on in Java Stream operations. Just install plugin called “Java Stream Debugger”. Once you have enabled this plugin you can simply debug your code. This plugin will bring a trace icon (Trace Current Stream Chain button).   Trace Current Stream Chain button Once you click on that icon you get the visualization of your stream pipeline.  For every stream operation, we have got a dedicated tab. you have to switch to the relevant tab to understand what it is doing. 
Read more

Creating Efficient Docker Images with Spring Boot 2.3

-
Image
Spring Boot 2.3 brings with it some interesting new features that can help you package up your Spring Boot application into Docker images. The first problem with common docker techniques is that the jar file is not unpacked. There’s always a certain amount of overhead when running a fat jar, and in a containerized environment this can be noticeable. It’s generally best to unpack your jar and run in an exploded form. The second issue with the file is that it isn’t very efficient if you frequently update your application. Docker images are built in layers, and in this case your application and all its dependencies are put into a single layer. Since you probably recompile your code more often than you upgrade the version of Spring Boot you use, it’s often better to separate things a bit more. If you put jar files in the layer before your application classes, Docker often only needs to change the very bottom layer and can pick others up from its cache. Two new features are introduced in Sp...
Read more

CRUD Goes Even Easier With JPABuddy

-
Image
Create, Read, Update and Delete are the four basic operations of persistence storage. We can say these operations collectively as an acronym CRUD. These operations can be implemented in JPA. JPA is a standard for ORM. It is an API layer that maps Java objects to the database tables.  ORM stands for Object Relational Mapping. It converts data between incompatible type systems in object-oriented programming languages.  JPA Buddy is an IntelliJ IDEA plugin that helps developers work efficiently. JPA Buddy is a tool that is supposed to become your faithful coding assistant for projects with JPA and everything related. It is an advanced plugin for IntelliJ IDEA intended to simplify and accelerate everything related to JPA and surrounding mainstream technology. In fact, you can develop an entire CRUD application or a simple microservice by spending nearly zero time writing boilerplate code. The video demonstrates the features of JPA Buddy by creating a simple CRUD application from ...
Read more