Redis Pop list item By numbers of items

Question

I have a distributed system where In one place i insert around 10000 items in a redis list then Call my multiple applications hook to process items. what i need is to Have some ListLeftPop type of methhod with numbers of items. It should remove items from the redis List and return to my calling application.

I am using Stackexchange.Resis.extension

My current method just for get (not pop) is

 public static List<T> GetListItemRange<T>(string key, int start, int chunksize) where T : class
        {
            List<T> obj = default(List<T>);
            try
            {
                if (Muxer != null && Muxer.IsConnected && Muxer.GetDatabase() != null)
                {
                    var cacheClient = new StackExchangeRedisCacheClient(Muxer, new NewtonsoftSerializer());
                    var redisValues = cacheClient.Database.ListRange(key, start, (start + chunksize - 1));
                    if (redisValues.Length > 0)
                    {
                        obj = Array.ConvertAll(redisValues, value => JsonConvert.DeserializeObject<T>(value)).ToList();
                    }
                }
            }
            catch (Exception ex)
            {
                Logger.Fatal(ex.Message, ex);
            }
            return obj;
        }

For Pop and get i have find a snippet

 var cacheClient = new StackExchangeRedisCacheClient(Muxer, new NewtonsoftSerializer());
                    var redisValues = cacheClient.ListGetFromRight<T>(key);

But it will only do for single item

Not all but by some configurable number. Let's say 100 at a time — Kamran Shahid, Jun 13 '16 at 04:45
Have tried var cacheClient = new StackExchangeRedisCacheClient(Muxer, new NewtonsoftSerializer()); var redisValues = cacheClient.ListGetFromRight(key); but it will only pop single item — Kamran Shahid, Jun 13 '16 at 05:51
Is it a queue or a stack? I mean lpush lpop (queue) or lpush rpop (stack). — Karthikeyan Gopall, Jun 13 '16 at 07:12

Karthikeyan Gopall · Accepted Answer · 2016-06-13T11:30:35.797

I assume that you are working on a queue, where you insert 1000 items at a single place and retrieve them at multiple place in the order at which it is inserted.

You can't achieve it with a single command but you can do it with 2 commands. You can write a lua script to make them atomic.

Lrange : http://redis.io/commands/lrange

Lrange list -100 -1

This will list you first 100 elements in the list. here the offset is -100. Note that this will return the items in the opposite order at which it is inserted. So you need to reverse the loop to ensure the queue mechanism.

Ltrim : http://redis.io/commands/ltrim

ltrim list 0 -101

This will trim the 1st 100 elements in the list. here 101 is n+1 so it must be 101. Here offset is 101

Writing them inside a lua block will ensure you the atomicity.

Let me give you a simple example.

You insert 100 elements in a single place.

lpush list 1 2 3 .. 100

You have multiple clients each trying to access this lua block. Say your n value is 5 here. 1st client gets in and gets first 5 elements inserted.

127.0.0.1:6379> lrange list -5 -1
1) "5"
2) "4"
3) "3"
4) "2"
5) "1"

You keep them in your lua object and delete them.

127.0.0.1:6379> LTRIM list 0 -6
OK

return them to your code, now the result you want is 1 2 3 4 5 but what you have got is 5 4 3 2 1. So you need to reverse the loop and perform the operation.

When the next client comes in it will get the next set of values.

127.0.0.1:6379> lrange list -5 -1
1) "10"
2) "9"
3) "8"
4) "7"
5) "6"

In this way you can achieve your requirement. Hope this helps.

EDIT:

Lua script:

local result = redis.call('lrange', 'list','-5','-1')
redis.call('ltrim','list','0','-6')
return result

Thanks Karthikeyan. My Analysis was same that i need two commands. And I might need single lua script to achieve it. I will try to make that single lua script which i can call from ScriptEvaluate. But I were looking for a solution directly from the api without need of the Lua script — Kamran Shahid, Jun 13 '16 at 08:07
Can you help me in a combined Lua script for those operations. Previously i have just a single experience of such script at http://stackoverflow.com/questions/34871567/redis-distributed-increment-with-locking — Kamran Shahid, Jun 13 '16 at 11:07

score 0 · Answer 2 · answered Jun 15 '16 at 04:54

Thanks Karthikeyan. My C# code with redis is as following

 public static RedisResult PopListItemRange(int chunksize, string key)
        {
            RedisResult valreturn = null;
            try
            {

                IDatabase db;
                if (Muxer != null && Muxer.IsConnected && (db = Muxer.GetDatabase()) != null)
                {
                    valreturn = db.ScriptEvaluate(@" local result = redis.call('lrange',KEYS[1],ARGV[1], '-1')
                                                          redis.call('ltrim',KEYS[1],'0',ARGV[2])
                                    return result", new RedisKey[] { key }, flags: CommandFlags.HighPriority, values: new RedisValue[] { -chunksize, -chunksize - 1 });
                }

            }
            catch (Exception ex)
            {
                Logger.Fatal(ex.Message, ex);
            }
            return valreturn;
        }

Now just doing a tweak that when the list is empty remove the redis Key

Redis Pop list item By numbers of items

2 Answers2

Linked