Thread in 'parking to wait' state on Redis dequeue

Question

I have a tomcat - spring4.2 application that runs multiple threads. Each thread dequeues from only one queue, however there are more than one threads assigned to a queue.

Things start fine, but after few hours / ~500k dequeue operations, I find threads dequeue at extremely slow rate.

In jvisualvm I see the threads in orange i.e. park The thread dump is as follows:

"EMLT_2" - Thread t@64
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <2cf42d7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
    at redis.clients.util.Pool.getResource(Pool.java:48)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86)
    at com.mycomp.sam.processors.SimpleDequeuer.dequeue(SimpleDequeuer.java:25)
    at com.mycomp.sam.processors.EMLT.run(EMLT.java:29)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"EMLT_1" - Thread t@63
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <2cf42d7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
    at redis.clients.util.Pool.getResource(Pool.java:48)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86)
    at com.mycomp.sam.processors.SimpleDequeuer.dequeue(SimpleDequeuer.java:25)
    at com.mycomp.sam.processors.EMLT.run(EMLT.java:29)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

The dequeue method is :

public String dequeue(String queue) {
        try (Jedis jedis = jedispool.getResource()) {
            List<String> str = jedis.blpop(10, queue);
            if(str!=null){
                return str.get(1);
            }
            else 
                return null; 
        }
    }

Will appreciate inputs. The app again works well for a while post restart. Pool conf:

<bean id="poolConfig" class="redis.clients.jedis.JedisPoolConfig">
    <property name="maxIdle" value="10" />
    <property name="maxTotal" value="70" />
    <property name="minIdle" value="10" />
</bean>
<bean id="jedispool" class="redis.clients.jedis.JedisPool">
    <constructor-arg name="poolConfig" ref="poolConfig" />
    <constructor-arg name="host" value="${REDIS_HOST}" />
    <constructor-arg name="port" value="6379" />
</bean>

Thought it was a connection leak, but try with resources shouldn't do that. — Kedar Parikh, May 21 '16 at 06:29
After all the research found that the pool was injected in one more class, long written and forgotten and it had a connection leak. silly. — Kedar Parikh, Jul 08 '16 at 07:25

score 2 · Accepted Answer · answered May 27 '16 at 00:32

2

It appears your jedis pool is running out of connections. Are you returning resources with returnResourceObject or returnResource? (I know both are deprecated, but they still work with the lates version of jedis).

Keep in mind that a jedis pool is not exactly like a DBCP/Apache Pool.

I had a similar issue and it ended when i called the methods above.

On the other hand, since Redis is pretty fast, maybe you should rethink your schema, and have very few threads calling blpop (one per queue) and the forward the values to the other threads, so those don't know about redis if they don't have to.

answered May 27 '16 at 00:32

Carlos Grappa

2,351
15
18

Even I thought so, but how do I verify that? I am not doing anything special apart from `try (Jedis jedis = jedispool.getResource()) {` Do I need to call these methods explicitly? – Kedar Parikh May 27 '16 at 07:13
At least in my case i had to, otherwise it wouldn't work. – Carlos Grappa May 27 '16 at 12:44
Thinking of switching to lettuce. `finally { if (jedis != null) { jedispool.returnResourceObject(jedis); } }` didnt help – Kedar Parikh Jun 02 '16 at 08:16
2

A couple of extra ideas you can try, call jedis.close() directly. And add this setting to your pool config maxWaitMillis: 100. The last one should mitigate the issue, but it still there. So maybe you have far to many requests por your pool size and you're getting swamped. – Carlos Grappa Jun 02 '16 at 13:50

Thread in 'parking to wait' state on Redis dequeue

1 Answers1