0

When I am trying to disable Spring Batch Metadata creation with the option spring.batch.initialize-schema=never and then I launch batch, nothing happen and the batch terminate immediately without running the related jobs. In the other hand when I am trying to enable the Metadata creation, the batch work fine, I am getting the classic SERIALIZED_CONTEXT field size error. I can't always save 4GB of data in the table when I execute the batch. How to disable definitively the Metadata creation, and have my batch still working?

Edit : I think I found a kind of solution to avoid this issue, and I would like to have your point of view. I am finally working with Metadata generation. The issue occurs when you have large set of data stored in your ExecutionContext you pass between Tasklets (we all know this is the reason). In my case it is an ArrayList of elements (POJO), retrieved from a CSV file with OpenCSV. To overcome this issue I have :

  • reduced the number of columns and lines in the ArrayList (because Spring Batch will serialize this ArrayList in the SERIALIZED_CONTEXT field. The more columns and lines you have the more you are sure to get this issue)
  • changed the type of the SERIALIZED_CONTEXT from TEXT to LONGTEXT
  • deleted the toString() method defined in the POJO (not sure it really helps)

But I am still wondering, what if you have no choice and you have to load all your columns, what is the best way to prevent this issue?

Harry Coder
  • 2,429
  • 2
  • 28
  • 32

1 Answers1

1

So this is not an issue with metadata generation but with passing a large amount of data between two steps.

what if you have no choice and you have to load all your columns, what is the best way to prevent this issue?

You can still load all columns but you have to reduce the chunk size. The whole point of chunk processing in Spring Batch is to not load all data in memory. What you can do in your case is to carefully choose a chunk size that fits your requirement. There is no recipe for choosing the correct chunk size (since it depends on the number of columns, the size of each column, etc), so you need to proceed in an empirical way.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • For the record, the following might help: https://stackoverflow.com/questions/2292667/how-can-we-share-data-between-the-different-steps-of-a-job-in-spring-batch – Mahmoud Ben Hassine Jul 29 '18 at 15:25