How to dynamically create a derived type in the Python C-API

Question

Assume we have the type Noddy as defined in the tutorial on writing C extension modules for Python. Now we want to create a derived type, overwriting only the __new__() method of Noddy.

Currently I use the following approach (error checking stripped for readability):

PyTypeObject *BrownNoddyType =
    (PyTypeObject *)PyType_Type.tp_alloc(&PyType_Type, 0);
BrownNoddyType->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE;
BrownNoddyType->tp_name = "noddy.BrownNoddy";
BrownNoddyType->tp_doc = "BrownNoddy objects";
BrownNoddyType->tp_base = &NoddyType;
BrownNoddyType->tp_new = BrownNoddy_new;
PyType_Ready(BrownNoddyType);

This works, but I'm not sure if it is The Right Way To Do It. I would have expected that I have to set the Py_TPFLAGS_HEAPTYPE flag, too, because I dynamically allocate the type object on the heap, but doing so leads to a segfault in the interpreter.

I also thought about explicitly calling type() using PyObject_Call() or similar, but I discarded the idea. I would need to wrap the function BrownNoddy_new() in a Python function object and create a dictionary mapping __new__ to this function object, which seems silly.

What is the best way to go about this? Is my approach correct? Is there an interface function I missed?

Update

There are two threads on a related topic on the python-dev mailing list (1) (2). From these threads and a few experiments I deduce that I shouldn't set Py_TPFLAGS_HEAPTYPE unless the type is allocated by a call to type(). There are different recommendations in these threads whether it is better to allocate the type manually or to call type(). I'd be happy with the latter if only I knew what the recommended way to wrap the C function that is supposed to go in the tp_new slot is. For regular methods this step would be easy -- I could just use PyDescr_NewMethod() to get a suitable wrapper object. I don't know how to create such a wrapper object for my __new__() method, though -- maybe I need the undocumented function PyCFunction_New() to create such a wrapper object.

Answer 1

I encountered the same problem when I was modifying an extension to be compatible with Python 3, and found this page when I was trying to solve it.

I did eventually solve it by reading the source code for the Python interpreter, PEP 0384 and the documentation for the C-API.

Setting the Py_TPFLAGS_HEAPTYPE flag tells the interpreter to recast your PyTypeObject as PyHeapTypeObject, which contains additional members that must also be allocated. At some point the interpreter attempts to refer to these extra members and, if you leave them unallocated, it will cause a segfault.

Python 3.2 introduced the C structures PyType_Slot and PyType_Spec and the C function PyType_FromSpec that simplify the creation of dynamic types. In a nutshell, you use PyType_Slot and PyType_Spec to specify the tp_* members of the PyTypeObject and then call PyType_FromSpec to do the dirty work of allocating and initialising the memory.

From PEP 0384, we have:

typedef struct{
  int slot;    /* slot id, see below */
  void *pfunc; /* function pointer */
} PyType_Slot;

typedef struct{
  const char* name;
  int basicsize;
  int itemsize;
  int flags;
  PyType_Slot *slots; /* terminated by slot==0. */
} PyType_Spec;

PyObject* PyType_FromSpec(PyType_Spec*);

(The above isn't a literal copy from PEP 0384, which also includes const char *doc as a member of PyType_Spec. But that member doesn't appear in the source code.)

To use these in the original example, assume we have a C structure, BrownNoddy, that extends the C structure for the base class Noddy. Then we would have:

PyType_Slot slots[] = {
    { Py_tp_doc, "BrownNoddy objects" },
    { Py_tp_base, &NoddyType },
    { Py_tp_new, BrownNoddy_new },
    { 0 },
};
PyType_Spec spec = { "noddy.BrownNoddy", sizeof(BrownNoddy), 0,
                      Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, slots };
PyTypeObject *BrownNoddyType = (PyTypeObject *)PyType_FromSpec(&spec);

This should do everything in the original code, including calling PyType_Ready, plus what is necessary for creating a dynamic type, including setting Py_TPFLAGS_HEAPTYPE, and allocating and initialising the extra memory for a PyHeapTypeObject.

I hope that's helpful.

Answer 2

I apologize up front if this answer is terrible, but you can find an implementation of this idea in PythonQt, in particular I think the following files might be useful references:

• PythonQtClassInfo.cpp
• PythonQtClassInfo.h
• PythonQtClassWrapper.cpp
• PythonQtClassWrapper.h

This fragment from PythonQtClassWrapper_init jumps out at me as being somewhat interesting:

static int PythonQtClassWrapper_init(PythonQtClassWrapper* self, PyObject* args, PyObject* kwds)
{
  // call the default type init
  if (PyType_Type.tp_init((PyObject *)self, args, kwds) < 0) {
    return -1;
  }

  // if we have no CPP class information, try our base class
  if (!self->classInfo()) {
    PyTypeObject*  superType = ((PyTypeObject *)self)->tp_base;

    if (!superType || (superType->ob_type != &PythonQtClassWrapper_Type)) {
      PyErr_Format(PyExc_TypeError, "type %s is not derived from PythonQtClassWrapper", ((PyTypeObject*)self)->tp_name);
      return -1;
    }

    // take the class info from the superType
    self->_classInfo = ((PythonQtClassWrapper*)superType)->classInfo();
  }

  return 0;
}

It's worth noting that PythonQt does use a wrapper generator, so it's not exactly in line with what you're asking for, but personally I think trying to outsmart the vtable isn't the most optimal design. Basically, there are many different C++ wrapper generators for Python and people use them for a good reason - they're documented, there are examples floating around in search results and on stack overflow. If you hand roll a solution for this that nobody's seen before, it'll be that much harder for them to debug if they run into problems. Even if it's closed-source, the next guy who has to maintain it will be scratching his head and you'll have to explain it to every new person who comes along.

Once you get a code generator working, all you need to do is maintain the underlying C++ code, you don't have to update or modify your extension code by hand. (Which is probably not too far away from the tempting solution you went with)

The proposed solution is an example of breaking the type-safety that the newly introduced PyCapsule provides a bit more protection against (when used as directed).

So, while its possible it might not be the best long term choice to implement derived/subclasses this way, but rather wrap the code and let the vtable do what it does best and when the new guy has questions you can just point him at the documentation for whatever solution fits best.

This is just my opinion though. :D

Answer 3

One way to try and understand how to do this is to create a version of it using SWIG. See what it produces and see if it matches or is done a different way. From what I can tell the people who have been writing SWIG have an in depth understanding of extending Python. Can't hurt to see how they do things at any rate. It may help you understand this problem.