How to add overloads to an existing API in PyQt

Question

There is an existing class whose __init__() already takes a varying number & types for its arguments. I wish to subclass to add a new argument. I do not know how I am intended to write the subclass's new __init__() definition.

I do not have the source for the existing base class (it's probably written in C++). help(QListWidgetItem) gives me:

class QListWidgetItem(sip.wrapper)
 |  QListWidgetItem(parent: QListWidget = None, type: int = QListWidgetItem.Type)
 |  QListWidgetItem(str, parent: QListWidget = None, type: int = QListWidgetItem.Type)
 |  QListWidgetItem(QIcon, str, parent: QListWidget = None, type: int = QListWidgetItem.Type)
 |  QListWidgetItem(QListWidgetItem)

My editor (PyCharm) recognises these and offers context-sensitive completion. It behaves as though they have been declared via @overload directives, and I wish to retain that.

Note that already not only is the number of arguments variable but also so are the types. For example, looking through all the overloads parameter #1 might be a QListWidget, a str, a QIcon or a QListWidgetItem, or not even supplied, and depending on that influences what the second argument can be, etc.

I wish to add an extra one:

MyListWidgetItem(text: str, value: QVariant, parent: QListWidget = None, type: int = QListWidgetItem.Type)

Note that my new QVariant argument is in second place, and I wish it to be positional not keyword-named.

So I need to recognise this new one when it's called; I need to pull out my new value: QVariant to set my new member variable, I also need to remove it before calling the base class constructor .

I know that for the declaration I will be adding an overload like:

class MyListWidgetItem(QListWidgetItem)
    @overload
    def __init__(self, text: str, value: QVariant, parent: QListWidget=None, type: int=QListWidgetItem):
        pass

(I assume that will leave the existing QListWidgetItem @overloads still available via my derived MyListWidgetItems?)

What about for the actual definition? What does it do and how should it be declared/written?

I need to recognise this new one when it's called; I need to pull out my new value: QVariant to set my variable, I also need to remove it before calling the base class constructor.

I can only guess: is it my job in order to recognise my case to write like:

if len(arguments) >= 2:
    if isinstance(arguments[0], str) and isinstance(arguments[1], QVariant):
        self.value = arguments[1]
        del arguments[1]

Then: Am I supposed to write the single __init__() definition (not @overload declarations) for my new sub-class along the lines of:

def __init__(self, *__args)
    ...
    super().__init__(*__args)

or with distinct, explicit, typed arguments along the lines of:

def __init__(self, arg1: typing.Union[QListWidget, str, icon, QListWidgetItem, None], arg2: typing..., arg3: typing..., arg4)
    ...
    super().__init__(arg1, arg2, arg3, arg4)

The latter looks complicated? Is the former approach declaring and working directly off *__args the best way to go?

[EDIT: If it makes any difference for producing some kind of solution, I am willing to make my new parameter optional via value: QVariant = .... Or, if the answer is, say, "You won't be able to do it quite your way because ..., but the better way to do this is to make it a named-keyword-only argument because then you can ...", or whatever, I would consider in that light.]

Answer 1

There's no one-size-fits-all answer, and pyQt (which is really a thin layer above Qt and exposes quite a few C++ idioms) is not necessarily representative of the most common use cases.

As a general rule, it's considered better practice to explicitely copy (and extend) the parent class's initializer prototype, so you (generic "you" -> anyone having to maintain the code) don't necessarily have to read the parent class doc etc to know what's expected.

Now in some cases where

• the parent class takes a lot of arguments
• and your subclass doesn't mess with those arguments
• and your subclass only wants to ADD arguments
• and it's ok for you / your team to add those arguments as keyword-only arguments or to force them to come before any of the parent's ones
• and it's ok for you / your team to give up on autodoc / type hints etc

then using *args and **kwargs is indeed a solution:

def __init__(self, my_positional_arg, *args, **kwargs)
    # use .pop() to avoid passing it to the parent
    my_own_kw_arg = kw.pop("my_own_kw_arg", "nothing")
    super().__init__(*args, **kwargs)

Note that you really want to support both *args and **kwargs in this case, as even required positional args can be passed as named arguments, and that's most often how they're actually passed when the class (or function FWIW) takes a lot of arguments (it's easier to remember names than positions...).

Answer 2

There are two somewhat separate issues here.

The first is specific to PyCharm and it's use of the typing module. It generates pyi files with stubs defining the APIs of various third-party libraries (such as PyQt) so that it can provide auto-completion and such like. In addition, it supports user-defined type-hints and pyi files, which are documented here: Type Hinting in PyCharm. However, since I am not a PyCharm user, I cannot give any practical advice on exactly how you should define your own overload stubs so that they augment the existing PyQt ones. I assume it must be possible, though.

The second issue concerns exactly how to implement function overloads for existing PyQt APIs. The short answer to this is that you can't: Python simply does not support overloads in the same way that C++ does. This is because Python is dynamically typed, so that type of overloading makes no sense there. However, it is possible to work around this in various ways to provide equivalent behaviour.

For your specific case, the simplest solution demands a small compromise. Your question states: "Note that my new QVariant argument is in second place, and I wish it to be positional not keyword-named". If you're willing to forgo this requirement, it makes things a lot easier, because you can then define your sub-class like this:

class MyListWidgetItem(QListWidgetItem):
    def __init__(self, *args, value=None, **kwargs):
        super().__init__(*args, **kwargs)

or like this:

class MyListWidgetItem(QListWidgetItem):
    def __init__(self, *args, **kwargs):
        value = kwargs.pop('value', None)
        super().__init__(*args, **kwargs)

These sub-classes will support all the existing PyQt overloads without ever needing to know exactly how they are defined, since you are simply passing on the arguments to the base implementation. It is entirely up to the user to supply the correct arguments, but since the base class is provided by PyQt, it will automatically raise a TypeError if the wrong ones are given. This all helps to keep the implementation very simple, but it does put a premium on documenting your APIs properly, given that the function signature itself provides little or no hint about what the correct arguments should be. However, if you can also find a way to utilise PyCharm's type-hinting support as suggested above, that should get you pretty close to a very simple, workable solution.

But what if you weren't willing to compromise? The immediate problem this raises can be seen if you consider this signature:

QListWidgetItem(parent: QListWidget = None, type: int = QListWidgetItem.Type)

This permits creating an item with no arguments. But that immediately clobbers any new overload which defines required arguments, since Python will raise a TypeError if they are missing when the constructor is called at runtime. The only way to work around this is to use an *args, **kwargs signature and then explicitly check the number and type of all the arguments in the body of the __init__. Effectively, this is what functools.singledispatch and third-party packages like multipledispatch do, only via decorators. This doesn't really by-pass the above problem, though - it just moves it elsewhere and saves you having to maintain a whole load of complicated boiler-plate code.

I'm not going to give any dispatch-style examples here: firstly because I have no idea how they will play out in PyCharm (or even PyQt, for that matter), and secondly because they have already been covered in more generic SO questions like this one: Python function overloading. My advice would be to start with the much simpler implementation given above, and then consider experimenting with the other approaches if you find you really need to add overloads with non-keyword arguments.

One final approach to consider is what might be called The Standard Kitchen-Sink Overload. In this approach, you simply forget about the signatures of the existing APIs and define your sub-class something like this:

class MyListWidgetItem(QListWidgetItem):
    def __init__(self, text='', value=None, parent=None, icon=None,
                       item=None, type=QListWidgetItem.Type):
        if item is not None:
            super().__init__(item)
        elif icon is not None:
            super().__init__(icon, text, parent, type)
        else:
            super().__init__(text, parent, type)

Or if you don't care about type and the copy-constructor:

class MyListWidgetItem(QListWidgetItem):
    def __init__(self, text='', value=None, parent=None, icon=None):
        if icon is not None:
            super().__init__(icon, text, parent)
        else:
            super().__init__(text, parent)

The vast majority of Python/PyQt code probably uses some variation of this kind of approach. Thus, practicality beats purity, I guess...