how do I properly inherit from a superclass that has a __new__ method?

Question

Let's assume we have a class 'Parent' , that for some reason has __new__ defined and a class 'Child' that inherits from it. (In my case I'm trying to inherit from a 3rd party class that I cannot modify)

class Parent:
    def __new__(cls, arg):
        # ... something important is done here with arg

My attempt was:

class Child(Parent):
    def __init__(self, myArg, argForSuperclass):
         Parent.__new__(argForSuperclass)
         self.field = myArg

But while

p = Parent("argForSuperclass")

works as expected

c = Child("myArg", "argForSuperclass")

fails, because 'Child' tries to call the __new__ method it inherits from 'Parent' instead of its own __init__ method.

What do I have to change in 'Child' to get the expected behavior?

Answer 1

Firstly, it is not considered best practice to override __new__ exactly to avoid these problems... But it is not your fault, I know. For such cases, the best practice on overriding __new__ is to make it accept optional parameters...

class Parent(object):
    def __new__(cls, value, *args, **kwargs):
        print 'my value is', value
        return object.__new__(cls, *args, **kwargs)

...so children can receive their own:

class Child(Parent):
    def __init__(self, for_parent, my_stuff):
        self.my_stuff = my_stuff

Then, it would work:

>>> c = Child(2, "Child name is Juju")
my value is 2
>>> c.my_stuff
'Child name is Juju'

However, the author of your parent class was not that sensible and gave you this problem:

class Parent(object):
    def __new__(cls, value):
        print 'my value is', value
        return object.__new__(cls)

In this case, just override __new__ in the child, making it accept optional parameters, and call the parent's __new__ there:

class Child(Parent):
    def __new__(cls, value, *args, **kwargs):
        return Parent.__new__(cls, value)
    def __init__(self, for_parent, my_stuff):
        self.my_stuff = my_stuff

Answer 2

Child doesn't call Parent's __new__ instead of its own __init__, it calls it's own __new__ (inherited from Parent) before __init__.

You need to understand what these methods are for and when Python will call them. __new__ is called to come up with an instance object, and then __init__ is called to initialize the attributes of that instance. Python will pass the same arguments to both methods: the arguments passed to the class to begin the process.

So what you do when you're inheriting from a class with __new__ is exactly what you do when inheriting any other method which has a different signature in the parent than you want it to have. You need to override __new__ to receive Child's arguments and call Parent.__new__ with the arguments it expects. Then you override __init__ entirely separately (though with the same argument list, and the same need to call Parent's __init__ with its own expected argument list).

There are two potential difficulties that could get in your way, however, because __new__ is for customising the process of obtaining a new instance. There's no reason it has to actually create a new instance (it could find one that already exists), and indeed it doesn't even have to return an instance of the class. This can lead to the following issues:

1. If __new__ returns an existing object (say, because Parent expects to be able to cache its instances), then you may find your __init__ being called on already existing objects, which can be very bad if your objects are supposed to have changeable state. But then, if Parent expects to be able to do this sort of thing it will probably expect a bunch of constraints on how it is used, and sub-classing it in ways that arbitrarily break those constraints (e.g. adding mutable state to a class that expects to be immutable so it can cache and recycle its instances) is almost certainly not going to work even if you can get your objects initialised correctly. If this sort of thing is going on, and there isn't any documentation telling you what you should be doing to subclass Parent, then the third-party probably doesn't intend you to be subclassing Parent, and things are going to be difficult for you. Your best bet though would probably be to try moving all of your initialisation to __new__ rather than __init__, ugly as that is.

2. Python only calls the __init__ method if the class' __new__ method actually returns an instance of the class. If Parent is using its __new__ method as some sort of "factory function" to return objects of other classes, then subclassing in a straightforward fashion is very likely to fail, unless all you need to do is change how the "factory" works.

Answer 3

As I understand it, when you call Child("myarg", "otherarg"), that actually means something like this:

c = Child.__new__(Child, "myarg", "otherarg")
if isinstance(c, Child):
    c.__init__("myarg", "otherarg")

You could:

1. Write an alternative constructor, like Child.create_with_extra_arg("myarg", "otherarg"), which instantiates Child("otherarg") before doing whatever else it needs to.

2. Override Child.__new__, something like this:

.

def __new__(cls, myarg, argforsuperclass):
    c = Parent.__new__(cls, argforsuperclass)
    c.field = myarg
    return c

I haven't tested that. Overriding __new__ can quickly get confusing, so it's best to avoid it if possible.

Answer 4

While brandizzi's answer works well for simple examples, I had to dig a little deeper in my own use case: How to properly inherit from a class that only uses __new__ and the Child class has their own properties and you need to perform some operations/checks on the given data before actually creating the new object (so before super().__new__).

In my case this issue came up using the Shapely 2.0 package, trying to inherit from a shapely.geometry.Polygon class. There's an issue thread on the Shapely Github, and a working solution with custom Child attributes: https://github.com/shapely/shapely/issues/1233#issuecomment-977837620

from shapely.geometry import Point

class PropertyPoint(Point):
    _id_to_attrs = {}
    __slots__ = Point.__slots__  # slots must be the same for assigning __class__ - https://stackoverflow.com/a/52140968

    def __init__(self, coord: Tuple[float, float], name: str):
        self._id_to_attrs[id(self)] = dict(name=name)

    def __new__(cls, coord: Tuple[float, float], name: str):
        point = super().__new__(cls, coord)
        point.__class__ = cls
        return point

    ...  # Irrelevant code 
 
   @property
    def name(self):
        return self._id_to_attrs[id(self)]['name']

    @name.setter
    def name(self, name):
        self._id_to_attrs[id(self)]['name'] = name

    ...  # Irrelevant code

For anyone who should come across the same issues, I hope you save some time with this solution.