Pass data from a C function via double pointer in Cython

Question

I want to use a small C routine with Cython. The C function itself is

#include <stdio.h>
#include "examples.h"

void add_array(int **io_array, int n) {
    int i;
    int *array;

    array = (int *) malloc(n * sizeof(int));

    for(i = 0; i < n; i++) {
       array[i] = i;
    }

    *io_array = array;
}

And the function prototype:

#ifndef EXAMPLES_H
#define EXAMPLES_H

void add_array(int **io_array, int n);

#endif

Now I want to use Cython to interface the C function with:

cdef extern from "examples.h":
    void add_array(int **io_array, int n)


import numpy as np

def add(arr):
    if not arr.flags['C_CONTIGUOUS']:
        arr = np.ascontiguousarray(arr, dtype=np.int32) 

    cdef int[::1] arr_memview = arr

    add_array(&arr_memview[0], arr_memview.shape[0])

return arr

When compiling it gives the error:

pyexamples.pyx:13:14: Cannot assign type 'int *' to 'int **'

What is the right way to interface this function?

I don't know enough Cython to tell you the right way to do this, but the problem is certainly that your C function is set up to both allocate and populate the (C) array. You are passing a pointer to the array contents, but the function expects to receive a pointer to a pointer that it can update with the *location* of the contents. — John Bollinger, Jul 26 '18 at 20:34

ead · Accepted Answer · 2019-05-02T20:44:13.617

It will not work out-of-the-box with numpy-arrays. You will have to make the memory management yourself, for example:

%%cython
from libc.stdlib cimport free
def doit():
    cdef int *ptr;
    add_array(&ptr, 5)
    print(ptr[4])
    free(ptr)   #memory management

The difference to your attempt: &arr_memview[0] is pointer to an integer array, but what you need for your function is a pointer to a pointer to an integer array - that is what &ptr is.

The problem with your function is, that it has too many responsibilities:

it allocates the memory
it initializes the memory

It would be easier, if add_array would only be doing the second part, i.e.

void add_array(int *io_array, int n) {
    int i;
    for(i = 0; i < n; i++) {
       io_array[i] = i;
    }
}

And thus any memory could be initialized (also memory which was not allocated with malloc).

However, it is possible to create a numpy-array using the returned pointer ptr, it is just less straight forward:

cimport numpy as np
import numpy as np

np.import_array()   # needed to initialize numpy-data structures

cdef extern from "numpy/arrayobject.h":
    void PyArray_ENABLEFLAGS(np.ndarray arr, int flags) #not include in the Cython default include

def doit():
    cdef int *ptr;
    add_array(&ptr, 5)

    # create numpy-array from data:
    cdef np.npy_intp dim = 5
    cdef np.ndarray[np.int32_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &dim, np.NPY_INT32, ptr)
    # transfer ownership of the data to the numpy array:
    PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
    return arr

The following is worth mentioning:

np.import_array() is needed to be able to use all of the numpy's functionality. Here is an example of what can happen, if np.import_array() isn't called.
After PyArray_SimpleNewFromData, the data itself isn't owned by the resulting numpy array, thus we need to enable the OWNDATA-flag, otherwise there will be a memory leak.
It is not obvious, that the resulting numpy-array can be responsible for freeing the data. For example instead of using malloc/free it could be using Python's memory allocator.

I would like to elaborate about point 3. above. Numpy uses a special function to allocate/deallocate memory for data - it is PyDataMem_FREE and uses system's free for it. So in your case (using system's malloc/free in add_array) everything is Ok. (PyDataMem_FREE should not be confused with PyArray_free, as I did in an earlier version of the answer. PyArray_free is responsible for freeing other elements (array itself, and dimension/strides data, not data-memory) of the numpy-array, see here and is different depending on Python version).

A more flexible/safe approach is to use PyArray_SetBaseObject as shown in this SO-post.

I used your suggestions and removed the memory management from the C function. Following your solution and [link] http://docs.cython.org/en/latest/src/userguide/memoryviews.html#pass-data-from-a-c-function-via-pointer solves my problem. — JaW., Jul 27 '18 at 12:27

Pass data from a C function via double pointer in Cython

1 Answers1