0

I understood that there are some limitations regarding the items processing order when it comes to using multi-threading.

From my understanding, when we configure a Step (read process, write) to use multi-threading (taskExecutor for example), we can't guarantee the items incoming order, since we don't know which thread will process first.

Is there a safe and a simple way to read and process a flat file (FlatFileItemReader) with the initial order of its items using multi-threading ?

Thank you

YellowStrawHatter
  • 856
  • 7
  • 16
  • 2
    Unless you are doing some academic exercise I doubt you will gain anything from reading with multiple threads. See https://stackoverflow.com/a/18972018/4110190 where @Gene explains why it might even be slower. – Jonas Pedersen Oct 05 '22 at 20:40
  • Hello Jonas, it's an entreprise scale batch processing project using spring batch, we need to keep the file lines initial order and at the same time we want to increase performance. If using multi-threading won't be faster when processing items then why is it provided by the spring batch API ? – YellowStrawHatter Oct 06 '22 at 07:07
  • I do not know the specific reason. But multi-threading in Spring Batch can indeed be faster when reading from multiple sources. But reading from the same physical source will need some form of synchronisation to not read randomly from the file. You could read the content of the file in one thread and then split the processing of parts in multiple threads. You should be able to control the ordering if you know which part of the content being processed by which thread. If the processing changes the order, then you could have some problems joining the result into one again. – Jonas Pedersen Oct 06 '22 at 11:43
  • I agree. My use case is getting more complicated, because I have to read a main file, then I need to open a second file and find some information (using some column value as a join key), the relation between the main file and the second one is 1..N, and the join key is a numeric value (both files are already sorted that way) .. so even the processing needs to respect the initial order. – YellowStrawHatter Oct 06 '22 at 14:50

0 Answers0