How to refactor method returning closure to be pickleable?

Question

My particular problem is Django related, but I rewrote the relevant code to general Python for better understanding.

import pickle


class FieldTracker(object):
    def patch_save(self, instance):
        original_save = instance.save

        def save(**kwargs):
            ret = original_save(**kwargs)

            # use properties of self, implement new stuff etc
            print 'Used patched save'

            return ret

        instance.save = save


class Model(object):
    name_field = 'joe'

    field_tracker = FieldTracker()

    def __init__(self):
        self.field_tracker.patch_save(self)

    def save(self):
        print 'Used default save'

model = Model()
model.save()  # Uses patched version of save

pickle.dumps(model)  # Fails

Model is representation of DB row. FieldTracker tracks changes to fields in Model (name_field in this case). FieldTracker needs to patch save method after Model is instantiated. Patched save is closure inside of patch_save as it uses properties from FieldTracker, calls methods from passed instance etc.

FieldTracker cannot be pickled as one if its methods contains closure. From what I tried, save cannot be moved to class level as I get TypeError: can't pickle instancemethod objects. When I tried to move patch_save to top level it produced the same exception as above code (surprise, surprise). Moving save to top level would probably mean usage of global variables, which I want to avoid (but I haven't actually tried it).

And the question is: Is it possible to refactor FieldTracker code to be pickleable or should I use different approach (like move overriden save to model mixin)?

This is real FieldTracker if anyone cares.

`instance.save` is a method of the object `instance`, right? The error is telling you that you can't pickle that. I think you'll need another pickleable class to call it for you, perhaps as `FieldTracker.save(instance)`. Then the first time you patch the save function, when the original save is the instancemethod, pass this other object instead of the instancemethod. — Steve Jessop, Oct 16 '13 at 14:56
I'm not sure if I understand what you mean. Could you try to write (pseudo)code into this gist describing approximate original scenario: https://gist.github.com/ondrowan/7012154? — Ondrej Slinták, Oct 16 '13 at 18:10
That's a different question. To fix that code, move the function `save` from the method `patch_save` to file level. You just have to believe the error messages. When it says, "can't pickle instancemethod object", that's because instance methods cannot be pickled. When it says, "cannot pickle function save, is not found as `__main__.save`", that's because functions cannot be pickled unless they're defined at module level (and btw the pickled form is just the name of the function, not the code). — Steve Jessop, Oct 16 '13 at 22:30
Ok, I've rewritten the question as it caused confusion. I hope it's clear why moving `save` to top level isn't possible (or is it?). If I moved it to top level I'd need to pass some variables (tracker instance, original save method) into it and I have no idea how. — Ondrej Slinták, Oct 17 '13 at 13:54
you *do* have an idea how. You wrote a class `PatchedSave` that was correct except it stored an instance method. Use exactly the same technique to write a callable class that can call the original `save` method, and store an instance of that class instead of storing an instancemethod. — Steve Jessop, Oct 17 '13 at 14:04

score 1 · Accepted Answer · answered Oct 19 '13 at 19:35

Why bother refactoring? I'm guessing your really interested in pickling the classes and instances as you wrote them, right? To do this, I'd use dill, which can pickle almost anything in python.

>>> import dill 
>>> class FieldTracker(object):
...   def patch_save(self, instance):
...     original_save = instance.save
...     def save(**kwargs):
...       ret = original_save(**kwargs)
...       print("Used patched save")
...       return ret
...     instance.save = save
... 
>>> class Model(object):
...   name_field = 'joe'
...   field_tracker = FieldTracker()
...   def __init__(self):
...     self.field_tracker.patch_save(self)
...   def save(self):
...     print("Used default save")
... 
>>> model = Model()
>>> model.save()
Used default save
Used patched save
>>> _model = dill.loads(dill.dumps(model))
>>> _model.save()
Used default save
Used patched save

Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

How to refactor method returning closure to be pickleable?

1 Answers1

Linked