57

I am looking for some clarification on the subject. In Kafka documentations I found the following:

Kafka only provides a total order over messages within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order over messages this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.

So here are my questions:

  1. Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition?

  2. Does it mean I need same amount of partitions as amount of consumers for the same group?

  3. How many consumers can read from one partition?

Also have some questions regarding relationship between keys and partitions with regard to API. I only looked at .net APIs (especially one from MS) but looks like the mimic Java API. I see when using a producer to send a message to a topic there is a key parameter. But when consumer reads from a topic there is a partition number.

  1. How are partitions numbered? Starting from 0 or 1?
  2. What exactly relationship between a key and partition? As I understand some function on key will determine a partition. is that correct?
  3. If I have 2 partitions in a topic and want some particular messages go to one partition and other messages go to another I should use a specific key for one specific partition, and the rest for another?
  4. What if I have 3 partitions and one type of messages to one particular partition and the rest to other 2?
  5. How in general I send messages to a particular partition in order to know for a consumer from where to read? Or I better off with multiple topics?

Thanks in advance.

Toni
  • 3,296
  • 2
  • 13
  • 34
Igor K.
  • 915
  • 2
  • 12
  • 22

2 Answers2

68

Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition?

Let's see the following properties of kafka:

  • each partition is consumed by exactly one consumer in the group
  • one consumer in the group can consume more than one partition
  • the number of consumer processes in a group must be <= number of partitions

With these properties, kafka is smartly able to provide both ordering guarantees and load balancing over a pool of consumer processes.

To answer your question, yes, in the context of the same group, if you want to have N consumers, you have to have at least N partitions.

Does it mean I need same amount of partitions as amount of consumers for the same group?

I think this has been explained in the first answer.

How many consumers can read from one partition?

The number of consumers that can read from one partition is always equal to the number of consumer groups subscribing to that topic.

Relationship between keys and partitions with regard to API

First, we must understand that the producer is responsible for choosing which record to assign to which partition within the topic.

Now, lets see how producer does so. First, lets see the class definition of ProducerRecord.java :

public class ProducerRecord<K, V> {

    private final String topic;
    private final Integer partition;
    private final Headers headers;
    private final K key;
    private final V value;
    private final Long timestamp;

}

Here, the field that we have to understand from the class is partition.

From the ProducerRecord docs,

  • If a valid partition number is specified, that partition will be used when sending the record.
  • If no partition is specified but a key is present a partition will be chosen using a hash of the key.
  • If neither key nor partition is present a partition will be assigned in a round-robin fashion.
jjlin
  • 4,462
  • 1
  • 30
  • 23
oblivion
  • 5,928
  • 3
  • 34
  • 55
  • 3
    The rule which partition message sent to is great. – g10guang Nov 14 '18 at 06:02
  • 2
    Thanks for explaining the difference between key and partition number. That threw me off while experimenting with the kafka-console-producer tool. Messages did not end up in the partition that I thought I had specified in what turns out to be the key value. – Christoph Jul 19 '19 at 14:57
  • Can two keys have same partition? How will message be stored in the same partition with different keys? – implosivesilence Mar 21 '21 at 07:12
  • So if we specify a key corresponding to our object (e.g. customer number) we get consistent assignments of objects to partitions and thus event ordering by key!! – Marc Sep 18 '21 at 08:17
  • Does this mean that if you where to say, use a correlationId as a key, you are going to be creating an ever growing number of partitions? Or are they instead just grouped into the same partition, any existing partition? – Douglas Gaskell Nov 25 '22 at 20:53
31

Partitions increase parallelism of Kafka topic. Any number of consumers/producers can use the same partition. Its up to application layer to define the protocol. Kafka guarantees delivery. Regarding the API, you may want to look at Java docs as they may be more complete. Based on my experience:

  1. Partitions start from 0
  2. Keys may be used to send messages to the same partition. For example hash(key)%num_partition. The logic is pluggable to Producer. https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/Partitioner.html
  3. Yes. but be careful not to end up with some key that will result in the "dedicated" partition. For this, you may want to have dedicated topic. For example, control topic and data topic
  4. This seems to be the same question as 3.
  5. I believe consumers should not make assumptions of the data based on partition. The typical approach is to have consumer group that can read from multiple partitions of a topic. If you want to have dedicated channels, it is better (safer/maintainable) to use separate topics.
Rambatino
  • 4,716
  • 1
  • 33
  • 56
YaRiK
  • 698
  • 6
  • 13