Python multiprocessing - mapping private method

Question

Generally I'm aware of pickle mechanism, but can't understand why this example:

from multiprocessing import Pool
class Foo:
    attr = 'a class attr'
    def __test(self,x):
        print(x, self.attr)

    def test2(self):
       with Pool(4) as p:
          p.map(self.__test, [1,2,3,4])

if __name__ == '__main__':
    f = Foo()
    f.test2()

complains about __test method?

return _ForkingPickler.loads(res)
AttributeError: 'Foo' object has no attribute '__test'

After changing def __test to def _test(one underscore) everything works fine. Do I miss any basics knowledge of pickleing or "private" methods?

Possible duplicate of [What is the meaning of a single and a double underscore before an object name?](https://stackoverflow.com/questions/1301346/what-is-the-meaning-of-a-single-and-a-double-underscore-before-an-object-name) — Markus Unterwaditzer, Aug 14 '19 at 15:09
@MarkusUnterwaditzer: This isn't simply a duplicate; this looks like a flaw in `pickle`, where it serializes on one end under the unmangled name, but tries to look up the mangled name on the other end. — ShadowRanger, Aug 14 '19 at 15:16

ShadowRanger · Accepted Answer · 2019-08-14T15:29:40.793

4

This appears to be a flaw in the name mangling magic. The actual name of a name-mangled private function incorporates the class name, so Foo.__test is actually named Foo._Foo__test, and other methods in the class just implicitly look up that name when they use self.__test.

Problem is, the magic extends to preserving the __name__ unmangled; Foo._Foo__test.__name__ is "__test". And pickle uses the __name__ to serialize the method. When it tries to deserialize it on the other end, it tries to look up plain __test, without applying the name mangling, so it can't find _Foo__test (the real name).

I don't think there is any immediate solution here aside from not using a private method directly (using it indirectly via another non-private method or global function would be fine); even if you try to pass self._Foo__test, it'll still pickle the unmangled name from __name__.

The longer term solution would be to file a bug on the Python bug tracker; there may be a clever way to preserve the "friendly" __name__ while still allowing pickle to seamlessly mangle as needed.

edited Aug 14 '19 at 15:29

answered Aug 14 '19 at 15:24

ShadowRanger

143,180
12
188
271

Thanks for detailed answer with link to name mailing magic. Now it looks cleaner for me. – How about nope Aug 14 '19 at 15:27
@Howaboutnope: I am filing a bug about this now, so don't feel obligated to do so. The underlying problem appears to be the `__reduce__` implementation of bound method objects, which serializes the method in a way that says "On unpickling, do `getattr(method.__self__, method.__func__.__name__)`" without accounting for `__name__` being unmangled. – ShadowRanger Aug 14 '19 at 15:53
I'd be grateful if you place link to this bug here, so others may also track it once they find this problem in a future. – How about nope Aug 14 '19 at 16:14
@Howaboutnope: Turns out an issue for this already existed: [Issue #33007: Objects referencing private-mangled names do not roundtrip properly under pickling](https://bugs.python.org/issue33007). – ShadowRanger Aug 16 '19 at 19:23

Python multiprocessing - mapping private method

1 Answers1

Linked