In spring batch how can we move the processed files to another folder, I am using MultiResourceItemReader and chunk processing

Question

In Spring batch application how can we move files from BatchFileDir folder to a destination folder as each file gets processed. There would be a bunch of files in BatchFileDir folder so I am using MultiResourceItemReader. I am also following the chunk based processing.

I have all batch files as :

    @Value("BatchFileDir/batchFile_*.csv")
    private Resource[] allBatchFiles;

The Item reader is:

    @Bean
    public MultiResourceItemReader<MyServiceRequest> multiResourceItemReader() {
    MultiResourceItemReader<MyServiceRequest> resourceItemReader = new 
    MultiResourceItemReader<MyServiceRequest>();
    resourceItemReader.setResources(allBatchFiles);
    FlatFileItemReader<Record> reader = new FlatFileItemReader<>();
    DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
    String[] tokens = {"id", "name", "author", "subject"};
    tokenizer.setNames(tokens);
    DefaultLineMapper< MyServiceRequest> lineMapper = new DefaultLineMapper<>();
    lineMapper.setLineTokenizer(tokenizer);
    lineMapper.setFieldSetMapper(new RecordFieldSetMapper());
    reader.setLinesToSkip(1);
    reader.setLineMapper(lineMapper);
    resourceItemReader.setDelegate(reader);
    return resourceItemReader;
}

and the Item processor is :

    @Bean
    public ItemProcessor< MyServiceRequest, MyServiceResponse> itemProcessor() {
    return new ModifiedItemProcessor();
    }

the ModifiedItemProcessor is :

    public class ModifiedItemProcessor implements ItemProcessor< MyServiceRequest, 
    MyServiceResponse > {
    
    public MyServiceResponse process(MyServiceRequest item) {
    // interact with other Microservices and get the response
    return response;

the step is :

       @Bean
       protected Step step(@Qualifier("itemProcessor") ItemProcessor<MyServiceRequest, 
       MyServiceResponse> processor, ItemWriter<MyServiceRequest> writer) {
       return stepBuilderFactory
            .get("myStep")
            .<MyServiceRequest, MyServiceResponse> chunk(99)
            .reader(multiResourceItemReader())
            .processor(processor)
            .writer(writer)
            .build();
}

Do you need to 1) move each file as soon as it gets processed, or 2) move all files once the step is finished? For 2), you can use a step listener or do it in a separate step. Have you tried one of these options? — Mahmoud Ben Hassine, Dec 07 '20 at 08:52
My priority is to move each file as processed. if that is not possible then atleast move all files once the step is finished. For this purpose I was trying to get the filename with getCurrentResource() and then move it to the required folder but that was giving NullPointerEception, then did not know if I can do something with Tasklet. Please do let me know how to use step listner in my scenario — SamA, Dec 07 '20 at 19:35
`For this purpose I was trying to get the filename with getCurrentResource() and then move it to the required folder`: don't use that method, it is deprecated for removal: https://github.com/spring-projects/spring-batch/issues/3776. I added an answer based on you comment. — Mahmoud Ben Hassine, Dec 07 '20 at 19:47

score 0 · Accepted Answer · answered Dec 07 '20 at 19:48

0

I don't see the added value of moving each file as soon as it gets processed, since all files will be moved after the step anyway. The listener approach should work, but there is an open issue on windows (https://github.com/spring-projects/spring-batch/issues/1500). Hence, I recommend using a separate step to move files after the main step. For that, you can inject the same resources in that step and move files in a tasklet.

answered Dec 07 '20 at 19:48

Mahmoud Ben Hassine

28,519
3
32
50

Thank you @Mahmoud, move files in a tasklet is some thing like https://stackoverflow.com/questions/52941094/how-to-move-files-to-archive-and-error-folders-after-processing – SamA Dec 08 '20 at 06:22
the added value of moving each file as soon as it gets processed is, for each file I wanted to say which records and how many records went wrong etc – SamA Dec 08 '20 at 15:28

In spring batch how can we move the processed files to another folder, I am using MultiResourceItemReader and chunk processing

1 Answers1