How to bind message types to Kafka consumers on a single consumer group?

Question

I have a producer producer who wants to send some_persistent_message to dbConsumer and some_Notification_message to notificationConsumer

Producer1 is sending the message with keys

db_key: some_persistent_message

notify_key:some_Notifiction_message

On the consumer side, I have a consumer group App1_group with the two consumers dbConsumer and notificationConsumer

At this point in time, my dbConsumer is always getting messages of some_Notification_message because of my consumer ending up owning a specific partition which is always getting notify_key:some_Notifiction_message

Is it possible to send some_persistent_message to dbConsumer and some_Notification_message to notification_consumer?

Is https://stackoverflow.com/questions/46492707/consumer-how-to-specify-partition-to-read-kafka helping? — Progman, Jun 08 '19 at 19:40
Why not use different topics and different consumer groups? One app can have multiple producers — OneCricketeer, Jun 09 '19 at 06:11
@cricket_007 Even multiple producers are not needed, one kafkaproducer object can produce to multiple topics. — hoodakaushal, Jun 09 '19 at 06:58

score 1 · Accepted Answer · answered Jun 09 '19 at 07:06

If you must mix multiple message types in a single topic (to maintain chronological order of the messages, for example), then I'd suggest having a single overloaded message type which can represent each of your disparate message types and which is able to answer the kind of message it holds. Then set up a separate consumer group for each message type and associated processing logic. Finally, modify the logic in each consumer to only process appropriate messages.

Having two different consumer groups assumes that you have roughly the same number of messages of each type, otherwise there could be a lot of wasted CPU. If one message type occurs orders of magnitudes more often that the other, you might be better off merging the two consumers into a single consumer with a branch to perform the appropriate processing.

Widely differing numbers of messages can also interfere with processing the two types of messages chronologically with two different consumer groups.

score 0 · Answer 2 · answered Jun 09 '19 at 06:57

The best way is to simply use KafkaConsumer#assign() and specify the proper partitions.

Alternatively if you are okay with twice the I/O, simply use two consumer groups - both consumers will get all the messages, and in your consumer logic you can simply skip the messages of unwanted type.

How to bind message types to Kafka consumers on a single consumer group?

2 Answers2