Intercept operator lookup on metaclass

Question

I have a class that need to make some magic with every operator, like __add__, __sub__ and so on.

Instead of creating each function in the class, I have a metaclass which defines every operator in the operator module.

import operator
class MetaFuncBuilder(type):
    def __init__(self, *args, **kw):
        super().__init__(*args, **kw)
        attr = '__{0}{1}__'
        for op in (x for x in dir(operator) if not x.startswith('__')):
            oper = getattr(operator, op)

            # ... I have my magic replacement functions here
            # `func` for `__operators__` and `__ioperators__`
            # and `rfunc` for `__roperators__`

            setattr(self, attr.format('', op), func)
            setattr(self, attr.format('r', op), rfunc)

The approach works fine, but I think It would be better if I generate the replacement operator only when needed.

Lookup of operators should be on the metaclass because x + 1 is done as type(x).__add__(x,1) instead of x.__add__(x,1), but it doesn't get caught by __getattr__ nor __getattribute__ methods.

That doesn't work:

class Meta(type):
     def __getattr__(self, name):
          if name in ['__add__', '__sub__', '__mul__', ...]:
               func = lambda:... #generate magic function
               return func

Also, the resulting "function" must be a method bound to the instance used.

Any ideas on how can I intercept this lookup? I don't know if it's clear what I want to do.

---

For those questioning why do I need to this kind of thing, check the full code here. That's a tool to generate functions (just for fun) that could work as replacement for lambdas.

Example:

>>> f = FuncBuilder()
>>> g = f ** 2
>>> g(10)
100
>>> g
<var [('pow', 2)]>

Just for the record, I don't want to know another way to do the same thing (I won't declare every single operator on the class... that will be boring and the approach I have works pretty fine :). I want to know how to intercept attribute lookup from an operator.

Answer 1

Some black magic let's you achieve your goal:

operators = ["add", "mul"]

class OperatorHackiness(object):
  """
  Use this base class if you want your object
  to intercept __add__, __iadd__, __radd__, __mul__ etc.
  using __getattr__.
  __getattr__ will called at most _once_ during the
  lifetime of the object, as the result is cached!
  """

  def __init__(self):
    # create a instance-local base class which we can
    # manipulate to our needs
    self.__class__ = self.meta = type('tmp', (self.__class__,), {})


# add operator methods dynamically, because we are damn lazy.
# This loop is however only called once in the whole program
# (when the module is loaded)
def create_operator(name):
  def dynamic_operator(self, *args):
    # call getattr to allow interception
    # by user
    func = self.__getattr__(name)
    # save the result in the temporary
    # base class to avoid calling getattr twice
    setattr(self.meta, name, func)
    # use provided function to calculate result
    return func(self, *args)
  return dynamic_operator

for op in operators:
  for name in ["__%s__" % op, "__r%s__" % op, "__i%s__" % op]:
    setattr(OperatorHackiness, name, create_operator(name))


# Example user class
class Test(OperatorHackiness):
  def __init__(self, x):
    super(Test, self).__init__()
    self.x = x

  def __getattr__(self, attr):
    print "__getattr__(%s)" % attr
    if attr == "__add__":
      return lambda a, b: a.x + b.x
    elif attr == "__iadd__":
      def iadd(self, other):
        self.x += other.x
        return self
      return iadd
    elif attr == "__mul__":
      return lambda a, b: a.x * b.x
    else:
      raise AttributeError

## Some test code:

a = Test(3)
b = Test(4)

# let's test addition
print(a + b) # this first call to __add__ will trigger
            # a __getattr__ call
print(a + b) # this second call will not!

# same for multiplication
print(a * b)
print(a * b)

# inplace addition (getattr is also only called once)
a += b
a += b
print(a.x) # yay!

Output

__getattr__(__add__)
7
7
__getattr__(__mul__)
12
12
__getattr__(__iadd__)
11

Now you can use your second code sample literally by inheriting from my OperatorHackiness base class. You even get an additional benefit: __getattr__ will only be called once per instance and operator and there is no additional layer of recursion involved for the caching. We hereby circumvent the problem of method calls being slow compared to method lookup (as Paul Hankin noticed correctly).

NOTE: The loop to add the operator methods is only executed once in your whole program, so the preparation takes constant overhead in the range of milliseconds.

Answer 2

The issue at hand is that Python looks up __xxx__ methods on the object's class, not on the object itself -- and if it is not found, it does not fall back to __getattr__ nor __getattribute__.

The only way to intercept such calls is to have a method already there. It can be a stub function, as in Niklas Baumstark's answer, or it can be the full-fledged replacement function; either way, however, there must be something already there or you will not be able to intercept such calls.

If you are reading closely, you will have noticed that your requirement for having the final method be bound to the instance is not a possible solution -- you can do it, but Python will never call it as Python is looking at the class of the instance, not the instance, for __xxx__ methods. Niklas Baumstark's solution of making a unique temp class for each instance is as close as you can get to that requirement.

Answer 3

It looks like you are making things too complicated. You can define a mixin class and inherit from it. This is both simpler than using metaclasses and will run faster than using __getattr__.

class OperatorMixin(object):
    def __add__(self, other):
        return func(self, other)
    def __radd__(self, other):
        return rfunc(self, other)
    ... other operators defined too

Then every class you want to have these operators, inherit from OperatorMixin.

class Expression(OperatorMixin):
    ... the regular methods for your class

Generating the operator methods when they're needed isn't a good idea: __getattr__ is slow compared to regular method lookup, and since the methods are stored once (on the mixin class), it saves almost nothing.

Answer 4

If you want to achieve your goal without metaclasses, you can append the following to your code:

def get_magic_wrapper(name):
    
    def wrapper(self, *a, **kw):
        print('Wrapping')
        res = getattr(self._data, name)(*a, **kw)
        return res
    return wrapper

_magic_methods = ['__str__', '__len__', '__repr__']
for _mm in _magic_methods:
    setattr(ShowMeList, _mm, get_magic_wrapper(_mm))

It reroutes the methods in _magic_methods to the self._data object, by adding these attributes to the class iteratively. To check if it works:

>>> l = ShowMeList(range(8))
>>> len(l)
Wrapping
8
>>> l
Wrapping 
[0, 1, 2, 3, 4, 5, 6, 7]
>>> print(l)
Wrapping
[0, 1, 2, 3, 4, 5, 6, 7]