Python C API Read a list of list, allocate memory and global Variables

Question

I am writing a module in order to optimize an operation but I have a few questions.

First

I need to read an array (list of list) from python to C, I've done that this way:

long i_k;
for (int k = 0; k < n; k++) {
  PyObject *first = PyList_GetItem(streamMatrixObj, k);
  for (int i = 0; i < n; i++) {
    PyObject *second = PyList_GetItem(first, i);
    i_k = PyLong_AsLong(second);
    f[k][i] = i_k;
  }
} 

The matrix always would be of n x n. Is there a better way of reading and storing the list of list into a C array?

Second

I am allocating memory for the C array like this:

void **createMatrix(uint rows, uint cols) {

  long **matrix;

  const size_t row_pointers_bytes = rows * sizeof(*matrix);
  const size_t row_elements_bytes = cols * sizeof(**matrix);
  matrix = PyObject_Malloc(row_pointers_bytes + rows * row_elements_bytes);

  if (!matrix) {
    PyErr_NoMemory();
    return NULL;
  }

  size_t i = 0;
  long *data = matrix + rows;
  for (i = 0; i < rows; i++)
    matrix[i] = data + i * cols;

  return matrix;
}

Should I use PyMem_Malloc or PyObject_Malloc?, I saw an explanation in this question

Is there any reason to use malloc over PyMem_Malloc?

But still don't know what is the appropriate function to call.

Third

The method of this module is going to be called many times, so, I would like to keep the C array in memory in order to avoid re-creating the array every time. For example, from python I would call the method myModule.deltaC(1,2,[1,2,3]) and that call will do a few operations with the matrix.

Could I initialize the Array before using deltaC function with for example myModule.initMatrix([[1,2,3],[1,2,3], [1,2,3]]) and keep it in memory until the program finish? The C array then would need to be a global variable in the module?

Update

I've tried initialize the C array in a global variable with module.initMatrix, but when call module.delta I get a SegFault when accessing the C Array, so I guess it is not possible or I am missing something.

Fourth

Is there a way to detect when the python program finish in order to call PyObject_free(array) to free the memory allocated for the C array?

Answer 1

First: That's pretty much how it should be done.

Second: It my understanding that you should you PyMem_Malloc/PyMem_Free here. PyObject_Malloc/PyObject_Free are intended for allocation of Python objects and is optimized for a lot of smaller objects. However, unless you allocate a lot of memory or make a very large number of allocations, it would not matter much.

Third & Fourth: you could use the Capsules API to store your C structures and pass them around API functions. Instead of storing a global matrix, you could have your initialization function (initMatrix) return a capsule containing the matrix and pass this capsule to the other functions (e.g. deltaC). When the matrix is no longer referenced, it will be automatically destroyed.

If you do not plan on storing anything but the matrix, a possible alternative is to use numpy arrays. These arrays are accessible directly via the numpy C API from C as a matrix. This would save you the need to call a Python C API function to retrieve each element of the matrix, to allocate memory yourself or to deallocate when it is destroyed.