How can I have a global C/C++ variable that is shared between Python C/C++ extensions?

Question

I have written a Python package almost completely in C++. The reason for doing so is because I want to manually wrap an existing C++ library, but that is irrelevant here.

This Python package consists of a number of different extension modules, all of which I compile with distutils in a 'setup.py' script. These extension modules can be interrelated, in which case I link them by passing the shared library to the Extension constructor. To be clear, suppose I have two Python C++ modules, A and B, where B uses functions defined in A. These normally compile into A.so and B.so. Since B uses functions defined in A, I compile the A module as usual then I pass ':A.so' as a library to the libraries keyword in the Extension constructor for the B module. (The ':' lets g++ deal with the fact that the library does not start with the usual 'lib' prefix.) This works fine for linking functions and classes.

My problem is as follows: I have defined some global C++ variables in A. While doing what I have described allows B to access functions in A, it actually seems to create a COPY of any global data defined in A. This is a real problem for me.

It seems to me that the issue is essentially similar to having global variables across shared libraries, as discussed here and elsewhere. That solution, and others I have found online, do not seem to solve the problem.

Any help would be very appreciated!

Edit: Forgot to mention that, yes, my global variables are declared as extern.

[mmap](http://pubs.opengroup.org/onlinepubs/009695399/functions/mmap.html)? — David Ranieri, Jun 08 '13 at 20:43
Maybe... Is mmap truly necessary? I think I have only one process, but I don't fully know what Python does underneath. — user2333829, Jun 08 '13 at 20:45
`Is mmap truly necessary?`, I don't know, its an option (Python is not my forte), also with Debian or Red Hat and derivates you can make use of /dev/shm — David Ranieri, Jun 08 '13 at 20:55
Anyone else able to help? While I appreciate the suggestions so far, I think I should be able to do this just with the right compiler flags. I'm really stuck... — user2333829, Jun 11 '13 at 08:14

score 2 · Answer 1 · answered Aug 14 '13 at 13:03

I guess the pythonic way is to let different extensions interact via a C API. Although I am not very experienced with c++ I guess it is not so much different from the solution in C. I would do it the following way:

define the global variable in module A
define a C API for module A, which contains a pointer to the global variable (and a macro for convenience if you wish).
load the C API in module B and access the global variable via the pointer.

Writing C APIs for python C extensions is a bit involved. (Read python docs on Python C API, if your are not familiar with it.) The minimal example of my proposed solution would look like this:

A.h

/* Header file for A module */

#ifndef A_MODULE_H
#define A_MODULE_H
#ifdef __cplusplus
extern "C" {
#endif

#define PyA_GET_X() (*x_ptr)
#define PyA_SET_X(x) (*x_ptr = x)

#ifdef A_MODULE
/* do nothing for this minimal example */
#else
static void **PyA_API;
#define x_ptr ((long *)PyA_API[0])

static int import_A(void)
{
    PyA_API = (void **)PyCapsule_Import("A._C_API", 0);
    return (PyA_API != NULL) ? 0 : -1;
}
#endif /* !defined(A_MODULE) */
#ifdef __cplusplus
}
#endif
#endif /* !defined(A_MODULE_H) */

A.c

#include <Python.h>
#define A_MODULE
#include "A.h"

long x = 0; /* here is the global variable */

static PyObject* 
set_x(PyObject *self, PyObject *args){
    if (!PyArg_ParseTuple(args, "l", &x)) return NULL;
    Py_RETURN_NONE;
}

static PyObject* 
get_x(PyObject *self, PyObject *args){
    return PyInt_FromLong(x);
}

static PyMethodDef methods[] = {
    {"set_x", (PyCFunction)set_x, METH_VARARGS, ""},
    {"get_x", (PyCFunction)get_x, METH_NOARGS, ""},
    {NULL, NULL, 0, NULL}
    };

#ifndef PyMODINIT_FUNC  /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC initA(void){
    PyObject *m = Py_InitModule3("A", methods, "");
    static void *PyA_API[1];
    PyA_API[0] = (void *)&x;

    PyObject *c_api_object = PyCapsule_New((void *)PyA_API, "A._C_API", NULL);
    if (c_api_object != NULL) PyModule_AddObject(m, "_C_API", c_api_object);
}

B.c

#include <Python.h>
#define B_MODULE
#include "A.h"

static PyObject* 
set_x(PyObject *self, PyObject *args){
    long y;
    if (!PyArg_ParseTuple(args, "l", &y)) return NULL;
    PyA_SET_X(y);
    Py_RETURN_NONE;
}

static PyObject* 
get_x(PyObject *self, PyObject *args){
    return PyInt_FromLong(PyA_GET_X());
}

static PyMethodDef methods[] = {
    {"set_x", (PyCFunction)set_x, METH_VARARGS, ""},
    {"get_x", (PyCFunction)get_x, METH_NOARGS, ""},
    {NULL, NULL, 0, NULL}
    };

#ifndef PyMODINIT_FUNC  /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC initB(void){
    import_A();
    Py_InitModule3("B", methods, "");

}

setup.py

from numpy.distutils.core import setup, Extension

setup(
    name="AB",
    ext_modules = [Extension('A', ['A.c']), Extension('B', ['B.c'])],
    )

and finally you will be able to read and modify x from both modules, either from C level or python. In python this will look like:

>>> import A, B
>>> A.set_x(1)
>>> B.get_x()
    1
>>> B.set_x(2)
>>> A.get_x()
    2

for access from C level use macros PyA_GET_X() and PyA_SET_X(x).

score 0 · Answer 2 · answered Jun 08 '13 at 20:58

0

If you can call from python code to c++ functions, just create a function that will access the global variable.

answered Jun 08 '13 at 20:58

eyalm

3,366
19
21

That's a possibility, but I want to access the C++ global variables within the C++ code for the Python extension modules. – user2333829 Jun 08 '13 at 20:59

How can I have a global C/C++ variable that is shared between Python C/C++ extensions?

2 Answers2