11

I have a series of caches which follow this pattern:

key_x_y = value

Like:

  'key_1_3' = 'foo'
  'key_2_5' = 'bar'
  'key_1_7' = 'baz'

Now I'm wondering how can I iterate over all keys to match pattern like key_1_* to get foo and baz using the native django cache.get()?

(I know that there are way, particularly for redis, that allow using more extensive api like iterate, but I'd like to stick to vanilla django cache, if possible)

Jason Aller
  • 3,541
  • 28
  • 38
  • 38
Jand
  • 2,527
  • 12
  • 36
  • 66
  • Are these keys available in `cache` object? Is `cache` object a dictionary? – AKS Mar 09 '16 at 14:51
  • Yes, cache object is a dict of strings saved in redis cache, and I'm going to find all values that match `key_1_*`. – Jand Mar 09 '16 at 14:52

3 Answers3

18

This is not possible using standard Django's cache wrapper. As the feature to search keys by pattern is a backend dependent operation and not supported by all the cache backends used by Django (e.g. memcached does not support it but Redis does). So you will have to use a custom cache wrapper with cache backend that supports this operation.

Edit: If you are already using django-redis then you can do

from django.core.cache import cache
cache.keys("foo_*")

as explained here.

This will return list of keys matching the pattern then you can use cache.get_many() to get values for these keys.

cache.get_many(cache.keys("key_1_*"))
Muhammad Tahir
  • 5,006
  • 1
  • 19
  • 36
  • Can you please elaborate with code, how can it be done in a custom wrapper? – Jand Mar 09 '16 at 15:04
  • Custom wrapper is an option when cache backend supports this feature but Django's cache wrapper does not. If backend does not support it then custom wrapper will do nothing. – Muhammad Tahir Mar 09 '16 at 15:27
  • 2
    As you yourself suggested in the question that it can be done using django-redis as Redis support this feature and django-redis is a custom wrapper that supports all the methods of Django wrapper plus some extra methods available only for Redis. You can look here http://niwinz.github.io/django-redis/latest/#_scan_delete_keys_in_bulk if you want to see how to get keys by pattern using django-redis. – Muhammad Tahir Mar 09 '16 at 15:29
  • Right Muhammad, so please use the most efficient way i.e `cache.get_many(cache.keys("key_1_*"))` and I'll accept the anser. – Jand Mar 09 '16 at 15:46
  • @MuhammadTahir can you explain time complexity. – Manoj Jadhav Mar 14 '17 at 12:49
  • 1
    @manojJadhav time complexity for searching key patterns is `O(N)` where `N` is the total numbers of keys stored in Redis database as mentioned here https://redis.io/commands/keys. Please also see the warning section on the referenced page where it states `should only be used in production environments with extreme care, it may ruin performance`. – Muhammad Tahir Mar 15 '17 at 07:20
2

If the cache has following entries:

cache = {'key_1_3': 'foo', 'key_2_5': 'bar', 'key_1_7': 'baz'}

You can get all the entries which has key key_1_*:

x = {k: v for k, v in cache.items() if k.startswith('key_1')}

Based on the documentation from django-redis

You can list all the keys with a pattern:

>>> from django.core.cache import cache
>>> cache.keys("key_1_*")
# ["key_1_3", "key_1_7"]

once you have the keys you can get the values from this:

>>> [cache.get(k) for k in cache.keys("key_1_*")]
# ['foo', 'baz']

You can also use cache.iter_keys(pattern) for efficient implementation.

Or, as suggested by @Muhammad Tahir, you can use cache.get_many(cache.keys("key_1_*")) to get all the values in one go.

AKS
  • 18,983
  • 3
  • 43
  • 54
  • Nice but the dictionary is save in redis cache. Sorry I should have mentioned this. – Jand Mar 09 '16 at 14:56
  • There are some [guidelines](http://michal.karzynski.pl/blog/2013/07/14/using-redis-as-django-session-store-and-cache-backend/) here on how to use Redis as cache backend. Have you set the cache to redis? – AKS Mar 09 '16 at 15:09
  • Yes, and I'm already using redis for sessions. That's why I don't want to break the cache by implementing third-party modules. – Jand Mar 09 '16 at 15:13
  • 2
    @AKS cache.get_many(cache.keys("key_1_*")) is a better option here. It will fetch all the keys in a single call instead of 'n' calls. – Muhammad Tahir Mar 09 '16 at 15:32
  • I agree that would be better. :) But, I just noticed that OP doesn't want to move away from vanilla django cache. – AKS Mar 09 '16 at 15:35
  • @AKS no away but move away vanilla django! thank you guys! – Jand Mar 09 '16 at 15:44
1

I saw several answers above mentioning django-redis.

Based on https://pypi.org/project/django-redis/

You can actually use delete_pattern() method

from django.core.cache import cache
cache.delete_pattern('key_1_*')
Rizky Arlin
  • 373
  • 1
  • 11