pickling lru_cached function on object

Question

As part of parallellizing some existing code (with multiprocessing), I run into the situation that something similar to the class below needs to be pickled.

Starting from:

import pickle
from functools import lru_cache

class Test:
    def __init__(self):
        self.func = lru_cache(maxsize=None)(self._inner_func)

    def _inner_func(self, x):
        # In reality this will be slow-running
        return x

calling

t = Test()
pickle.dumps(t)

returns

_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000190454A7AC8>: it's not the same object as __main__.Test._inner_func

which I don't really understand. By the way, I also tried a variation where the name of _inner_func was func as well, that didn't change things.

Looks like the same issue of using pickle with decorators: https://stackoverflow.com/questions/52185507/pickle-and-decorated-classes-picklingerror-not-the-same-object — piwai, Jan 25 '19 at 10:34
Thank you, I agree. If you post this as an answer, I'll mark that one as the solution — Bram, Jan 27 '19 at 08:28

score 1 · Answer 1 · answered Sep 20 '22 at 10:51

If anybody is interested, this can be solved by using getstate and setstate like this:

from functools import lru_cache
from copy import copy


class Test:
    def __init__(self):
        self.func = lru_cache(maxsize=None)(self._inner_func)

    def _inner_func(self, x):
        # In reality this will be slow-running
        return x

    def __getstate__(self):
        result = copy(self.__dict__)
        result["func"] = None
        return result

    def __setstate__(self, state):
        self.__dict__ = state
        self.func = lru_cache(maxsize=None)(self._inner_func)

score 0 · Accepted Answer · answered Jan 30 '19 at 10:36

0

As detailled in the comments, the pickle module has issues when dealing with decorators. See this question for more details:

Pickle and decorated classes (PicklingError: not the same object)

answered Jan 30 '19 at 10:36

piwai

158
2
10

score 0 · Answer 3 · answered May 05 '19 at 13:19

Use methodtools.lru_cache not to create a new cache function in __init__

import pickle
from methodtools import lru_cache

class Test:
    @lru_cache(maxsize=None)
    def func(self, x):
        # In reality this will be slow-running
        return x

if __name__ == '__main__':
    t = Test()
    print(pickle.dumps(t))

It requires to install methodtools via pypi:

pip install methodtools

pickling lru_cached function on object

3 Answers3