3

I used the PyTorch saving method to serialize a bunch of essential objects. Among those, there was one class referencing a private method inside the __init__ of that same class. Now, after the serialization, I can't deserialize (unpickle) files because the private method is not accessible outside the class. Any idea how to solve or bypass it? I need to recover the data saved into the attributes of that class.

  File ".conda/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-a5666d77c70f>", line 1, in <module>
    torch.load("snapshots/model.pth", map_location='cpu')
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
    result = unpickler.load()
AttributeError: 'Trainer' object has no attribute '__iterator'
  • EDIT-1:

Here there is a piece of code that will generate the problem I’m facing right now.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private  # buggy

    def __private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")

However, if you remove the private attribute from the method, you won’t get any error.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.private  # not buggy

    def private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
user3108967
  • 43
  • 1
  • 15

2 Answers2

1

This question is similar to Python multiprocessing - mapping private method, but can not be marked as duplicate because of the open bounty.

The issue arises from this open issue on the Python bug tracker: Objects referencing private-mangled names do not roundtrip properly under pickling, and is related to the way pickle handles name-mangling. More details on this answer: https://stackoverflow.com/a/57497698/6352677.

At this point, the only workaround is not using private methods in __init__.

Keldorn
  • 1,980
  • 15
  • 25
0

This problem is due to name mangling — where the interpreter changes the name of the variable in below way which makes it harder to create collisions when the class is extended later. where

self.__private

has been changed to (self._className__privateMethodName)

self._Test__private

As name mangling is not applied with dunder, where a name has to start and end with double underscores.

So, to avoid name mangling add two more underscores at the end.

Below snippet should solve your problem.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private__

    def __private__(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
Keldorn
  • 1,980
  • 15
  • 25
anil kumar
  • 774
  • 7
  • 7
  • OP's question is about serialization **with private method** in `__init__`, aka with name mangling. Here `__private__` has no name mangling, so is not private, despite the name. – Keldorn Apr 18 '20 at 04:27
  • If we have to use only `__private` method there is no way we can implement above code with serialization. – anil kumar Apr 18 '20 at 07:18
  • Right. So your answer to the question is "it cannot be done with private methods", which is my answer as well. But it is not stated explicitly in your answer, instead you say "add a trailing __", which I find obfuscating. If you rename the __private variable to make it public, let's be clear and call it public, not a misleading __private__ (which is public, and a special method / dunder -- for which reason?). – Keldorn Apr 18 '20 at 10:05