Redis Out of Memory Exceptions, but still have plenty of memory

Question

I'm using the StackeExchange.Redis project to interact with Redis in our .NET Core C# project.

Under heavy load, our Redis connections will begin to fail with the following exception:

StackExchange.Redis.RedisServerException: OOM command not allowed when used memory > 'maxmemory'

The problem is that we have a ridiculous amount of free memory left. We're using Elasticache, so it's easy to lookup:

We can also connect to Elasticache through a shell, and see that there is memory avaialable, and interact with it just fine.

This is the code I used as a layer over the Connection information.

    public class RedisTimeConnectionManager : IRedisConnectionManager
{
    // More info about the Lazy<> pattern https://stackoverflow.com/questions/28792196/how-does-connectionmultiplexer-deal-with-disconnects
    // Additional information about the multiplexer: https://github.com/StackExchange/StackExchange.Redis/blob/master/docs/Basics.md
    private static Lazy<ConnectionMultiplexer> RedisConnectionMultiplexer = new Lazy<ConnectionMultiplexer>(() =>
    {
        return ConnectionMultiplexer.Connect(ConnectionString);
    });

    private static string ConnectionString { get; set; }

    public RedisTimeConnectionManager(string connectionString)
    {
        ConnectionString = connectionString;
    }

    public ConnectionMultiplexer GetConnectionMultiplexer()
    {
        return RedisConnectionMultiplexer.Value;
    }

    public IDatabase GetDatabaseConnection()
    {
        return RedisConnectionMultiplexer.Value.GetDatabase();
    }
}

I then pass this Connection layer to my redis "time" manager. This is the code that is throwing the OOM error:

public class TimeRedisManager : ITimeRedisManager
{
    private IRedisConnectionManager RedisConnectionManager { get; }

    public TimeRedisManager(IRedisConnectionManager redisConnectionManager)
    {
        RedisConnectionManager = redisConnectionManager;
    }

    public async Task<RedisUserTimelineGetValueDto> GetValueAsync(string id)
    {
        string key = $"time:{id}";

        HashEntry[] entries = await RedisConnectionManager.GetDatabaseConnection().HashGetAllAsync(key);

        // Parse and return values...
    }
}

Because Elasticache has over 7.5GB free of memory, and because I can interact with it through a shell, I'm assuming it's either the StackExchange.Redis library, or an issue with connection management in my code.

.NET CORE 2.1 StackExchange.Redis v 2.0.513

One last important thing - when this exception happens, it keeps happening. Restarting the services that interact with Redis does nothing. Only restarting the Elasticache nodes solve the problem.

score 3 · Accepted Answer · answered Oct 25 '18 at 21:34

Redis could take 2 times the memory required by data stored in it.

Read more here : https://redis.io/topics/admin

If you are using Redis in a very write-heavy application, while saving an RDB file on disk or rewriting the AOF log Redis may use up to 2 times the memory normally used. The additional memory used is proportional to the number of memory pages modified by writes during the saving process, so it is often proportional to the number of keys (or aggregate types items) touched during this time. Make sure to size your memory accordingly.

So, if the data stored in Redis takes 8 Gb of space, under heavy load Redis may consume 16 Gbs. You may have to tune the memory accordingly if that's the case.

This seems like the correct answer. The Elasticache instance is a write heavy instance, and after looking at our Cloud metrics, the crashing starts to happen after very heavy write-loads, and after the free-able RAM starts to exponentially dip. — contactmatt, Oct 29 '18 at 15:25

Redis Out of Memory Exceptions, but still have plenty of memory

1 Answers1

Linked