Python 2.7 and 3.8.0 compatible django-redis serializer

Question

I Have two server running one is running on python2.7 and another on python3.8.

for both the servers we have the common Django-cache server.

python2.7 server is setting up cache in the Django-cache and python3.8 trying to read this throwing an error saying

File "/usr/local/bin/python-3.8.0/lib/python3.8/site-packages/django_redis/serializers/pickle.py", line > 35, in loads return pickle.loads(value) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 1: ordinal not in range(128)

i already read the below post with the same question link

my goal here is to able to read and from both versions of python to the common django-cache.

score 0 · Answer 1 · answered Jan 28 '20 at 17:47

Default str encoding in python 2.7 is ASCII while the default for 3.x is utf-8 so you need to consider this.

If you look closely it's not Redis issue, it's encoding issue, hence the error message:

UnicodeDecodeError: 'ascii' codec can't decode byte

To solve this problem you have to set the default encoding to utf-8 in your python 2.7 program.

import sys  

reload(sys)  
sys.setdefaultencoding('utf8')

There are more details about the python encoding conversion issue here: How to fix UnicodeDecoderError

while reading from the cache itself it reading as a byte non-ASCII character and it is checking format into an int(value) this is what raising an error. — i_am_deesh, Jan 29 '20 at 04:43

score 0 · Accepted Answer · answered Jan 29 '20 at 19:18

The solution which worked for me is we implemented a custom serializer and adder that in the redis_cache config.

import six
from django.utils.encoding import force_bytes
from django_redis.serializers.pickle import PickleSerializer

try:
    import cPickle as pickle
except ImportError:
    import pickle


class CcustomPickleSerializer(PickleSerializer):

    def loads(self, value):
        if six.PY3:
            return self._loads_py3(value)
        return super().loads(force_bytes(value))

    def _loads_py3(self, value):
        return pickle.loads(
            force_bytes(value),
            fix_imports=True,
            encoding='latin1'
        )

if you use the encoding method as 'bytes'. you can get bytes else you will get a string.

and the below-mentioned line in the cache config inside the settings.py.

'SERIALIZER':'file.location.CustomPickleSerializer' in.


CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/0',
        'OPTIONS': {
            'KEY_PREFIX': 'personify',
            'SERIALIZER':'file.location.CustomPickleSerializer',
            'PARSER_CLASS': 'redis.connection.HiredisParser',  # Hiredis is WAY faster than redis-py
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'CONNECTION_POOL_KWARGS': {
                'max_connections': 100
            },
            'PICKLE_VERSION': 2,  # Make sure we're using v2 of pickle, which is pretty efficient.
            'SOCKET_CONNECT_TIMEOUT': 5,  # Seconds (timeout for the connection to be established).
            'SOCKET_TIMEOUT': 5,  # Seconds (timeout for read and write operations after the connection is established).
            'IGNORE_EXCEPTIONS': False  # Set to True to act like memcached - i.e. don't raise errors.
        }
    }
}

Python 2.7 and 3.8.0 compatible django-redis serializer

2 Answers2