Spring Batch : Not all threads are running from the threadpool

Question

I have a spring batch application which reads records from DB table and makes REST API calls in the writer class to get the data to be cached. However I observe that not all threads from the thread pool are running. They just run in the batches of 4-5 due which application takes 5 hours to make 120K calls. Following is the spring batch context

Task Executor:

<bean id="myTaskExecutor"
    class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="corePoolSize" value="20"/>
    <property name="maxPoolSize" value="20"/>
</bean>

Batch Job:

<batch:job id="myBatchJob">
    <batch:step id="loadData">
        <batch:tasklet task-executor="myTaskExecutor"
            transaction-manager="batchTransactionManager">
            <batch:chunk reader="myReader" writer="myLoader"
                commit-interval="250">
            </batch:chunk>
            <batch:listeners>
                <batch:listener ref="batchStepListener" />
            </batch:listeners>
        </batch:tasklet>
    </batch:step>       
</batch:job>

Reader config: Note that the dataSource has 20 min/max connections

<batch:job id="myBatchJob">
    <batch:step id="loadLei">
        <batch:tasklet task-executor="myTaskExecutor"
            transaction-manager="batchTransactionManager">
            <batch:chunk reader="myReader" writer="myLoader"
                commit-interval="250">
            </batch:chunk>
            <batch:listeners>
                <batch:listener ref="batchStepListener" />
            </batch:listeners>
        </batch:tasklet>
    </batch:step>       
</batch:job>    

<bean id="myReader"
    class="org.springframework.batch.item.database.JdbcPagingItemReader">
    <property name="dataSource" ref="myDataSource" />
    <property name="queryProvider" ref="myQueryProvider" />
    <property name="pageSize" value="1000" />
    <property name="rowMapper" ref="myRowMapper" />
    <property name="saveState" value="false" />
</bean>

<bean id="myQueryProvider"
    class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
    <property name="dataSource" ref="myDataSource" />
    <property name="selectClause" value="SELECT * " />
    <property name="fromClause" value="MYTABLE"/>
    <property name="sortKeys">
        <map>
            <entry key="MYCOLUMN" value="ASCENDING"></entry>
        </map>
    </property> 
</bean>

<bean id="myRowMapper"
    class="com.my.RequestRowMapper">
</bean>

Have you read [the documentation](https://docs.spring.io/spring-batch/trunk/reference/html/scalability.html#multithreadedStep)? This clearly states that the default limit is 4 threads. — M. Deinum, Sep 20 '17 at 10:52
Thanks a lot. All these days I have been overlooking this configuration. It helped. — xjava, Sep 20 '17 at 11:58

score 0 · Answer 1 · answered Mar 20 '18 at 02:33

you need to set the throttle limit to your job. The default throttle limit is 4. in your case even though your thread pool has free threads available the spring batch will use only 4 threads.

take a look at this Spring batch corePoolSize VS throttle-limit

Spring Batch : Not all threads are running from the threadpool

1 Answers1