1

Context: invoicing system, the sent invoices must have consecutive numbers.

Each invoice has a unique invoice number, for sake of simplicity let's say they are I1, I2, I3, and so on. So, the first invoice in the system has the number I1, and it gets incremented for every next invoice. Then, each invoice is being produced in a Kafka topic.

So, one could always calculate the number for the next invoice only by the contents of this topic, right? (count of invoices in the topic + 1 = next number) We could call such a system event-sourced then.

But how do you achieve this? For me, this seems like a circular data flow: in order to produce into the topic, I first need to ensure that I consumed the same whole topic at another place.

Am I getting something wrong about event streaming or is it just impossible with Kafka?

Invoices are always being assigned the number and sent one-by-one, not in parallel.

Eduard Kukuy
  • 11
  • 3

1 Answers1

0

Producers shouldn't care about what has (or hasn't) been consumed.

You seem to simply need to ensure that the producer has acks=1, meaning the broker accepted the message, and that you have one partition to ensure complete ordering.

If you need to ensure atomically increasing values across distributed threads/processes, then you'll want to save off that number somewhere else rather than rely on the state of the topic (for example, a Zookeeper lock).

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • > Producers shouldn't care about what has (or hasn't) been consumed. jep, that's true. sorry, how exactly does `acks=1` solve the challenge? I imagine it this way: after the invoice was produced, we want to query the next consecutive number. For that, the topic gets consumed and written into a database (or maybe ksql comes into play). It is possible that at the moment of querying for the next invoice the latest produced has not arrived into the database yet, so a duplicate number gets produced. – Eduard Kukuy Feb 05 '21 at 00:49
  • As far as I understand it, `acks=1` ensures the absence of gaps in consecutive numbers, which is an important guarantee. My current idea is to save off that number in a relational DB (where I can increment it atomically within a transaction), and, at the start of the service, consume the entire topic and initialize the number in the database based on the contents of the topic. Would it count as event sourcing? – Eduard Kukuy Feb 05 '21 at 00:50
  • If you have acks=-1, then there's no guarantee numbers would be consecutive or in order. If you have acks=1 or all, then it doesn't prevent gaps or prevent duplicates, it simply ensures data gets written. And I don't think consuming an entire topic before starting a producer is a good idea, but if you think it's necessary, then go ahead – OneCricketeer Feb 05 '21 at 16:23
  • 1
    In a sense, it's event sourcing, but there are datastores that are much more suited to event sourcing than Kafka. – Levi Ramsey Feb 06 '21 at 00:19