How does extending classes (Monkey Patching) work in Python?

Question

class Foo(object):
  pass

foo = Foo()
def bar(self):
  print 'bar'

Foo.bar = bar
foo.bar() #bar

Coming from JavaScript, if a "class" prototype was augmented with a certain attribute. It is known that all instances of that "class" would have that attribute in its prototype chain, hence no modifications has to be done on any of its instances or "sub-classes".

In that sense, how can a Class-based language like Python achieve Monkey patching?

Answer 1

The real question is, how can it not? In Python, classes are first-class objects in their own right. Attribute access on instances of a class is resolved by looking up attributes on the instance, and then the class, and then the parent classes (in the method resolution order.) These lookups are all done at runtime (as is everything in Python.) If you add an attribute to a class after you create an instance, the instance will still "see" the new attribute, simply because nothing prevents it.

In other words, it works because Python doesn't cache attributes (unless your code does), because it doesn't use negative caching or shadowclasses or any of the optimization techniques that would inhibit it (or, when Python implementations do, they take into account the class might change) and because everything is runtime.

Answer 2

I just read through a bunch of documentation, and as far as I can tell, the whole story of how foo.bar is resolved, is as follows:

• Can we find foo.__getattribute__ by the following process? If so, use the result of foo.__getattribute__('bar').
  • (Looking up __getattribute__ will not cause infinite recursion, but the implementation of it might.)
  • (In reality, we will always find __getattribute__ in new-style objects, as a default implementation is provided in object - but that implementation is of the following process. ;) )
  • (If we define a __getattribute__ method in Foo, and access foo.__getattribute__, foo.__getattribute__('__getattribute__') will be called! But this does not imply infinite recursion - if you are careful ;) )
• Is bar a "special" name for an attribute provided by the Python runtime (e.g. __dict__, __class__, __bases__, __mro__)? If so, use that. (As far as I can tell, __getattribute__ falls into this category, which avoids infinite recursion.)
• Is bar in the foo.__dict__ dict? If so, use foo.__dict__['bar'].
• Does foo.__mro__ exist (i.e., is foo actually a class)? If so,
  • For each base-class base in foo.__mro__[1:]:
    • (Note that the first one will be foo itself, which we already searched.)
    • Is bar in base.__dict__? If so:
      • Let x be base.__dict__['bar'].
      • Can we find (again, recursively, but it won't cause a problem) x.__get__?
        • If so, use x.__get__(foo, foo.__class__).
        • (Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
        • Otherwise, use x.
• For each base-class base of foo.__class__.__mro__:
  • (Note that this recursion is not a problem: those attributes should always exist, and fall into the "provided by the Python runtime" case. foo.__class__.__mro__[0] will always be foo.__class__, i.e. Foo in our example.)
  • (Note that we do this even if foo.__mro__ exists. This is because classes have a class, too: its name is type, and it provides, among other things, the method used to calculate __mro__ attributes in the first place.)
  • Is bar in base.__dict__? If so:
    • Let x be base.__dict__['bar'].
    • Can we find (again, recursively, but it won't cause a problem) x.__get__?
      • If so, use x.__get__(foo, foo.__class__).
      • (Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
      • Otherwise, use x.
• If we still haven't found something to use: can we find foo.__getattr__ by the preceding process? If so, use the result of foo.__getattr__('bar').
• If everything failed, raise AttributeError.

bar.__get__ is not really a function - it's a "method-wrapper" - but you can imagine it being implemented vaguely like this:

# Somewhere in the Python internals
class __method_wrapper(object):
    def __init__(self, func):
        self.func = func
    def __call__(self, obj, cls):
        return lambda *args, **kwargs: func(obj, *args, **kwargs)
        # Except it actually returns a "bound method" object
        # that uses cls for its __repr__
    # and there is a __repr__ for the method_wrapper that I *think*
    # uses the hashcode of the underlying function, rather than of itself,
    # but I'm not sure.

# Automatically done after compiling bar
bar.__get__ = __method_wrapper(bar)

The "binding" that happens within the __get__ automatically attached to bar (called a descriptor), by the way, is more or less the reason why you have to specify self parameters explicitly for Python methods. In Javascript, this itself is magical; in Python, it is merely the process of binding things to self that is magical. ;)

And yes, you can explicitly set a __get__ method on your own objects and have it do special things when you set a class attribute to an instance of the object and then access it from an instance of that other class. Python is extremely reflective. :) But if you want to learn how to do that, and get a really full understanding of the situation, you have a lot of reading to do. ;)