I am using Pybind11 and trying to use OpenMP in it. I call a c++ function from Python using the PyBind interpreter and GIL, then I compute a multithreaded for loop with OpenMP in c++, in which I call within each thread a python function. To do so, I first limit the scope of the GIL with py::gil_scoped_release release; going until the parallel region and then recover it outside the parallel region with `py::gil_scoped_acquire acquire;
The problem is inside the multithreaded for loop. I need to create an interpreter for each thread. My question is: How to do it ?
I have first thought about using the py::scoped_interpreter guard{}, but in the documentation, it said that:
Creating two concurrent scoped_interpreter guards is a fatal error. So is >calling initialize_interpreter() for a second time after the interpreter has already been initialized.
Do not use the raw CPython API functions Py_Initialize and Py_Finalize as these do not properly handle the lifetime of pybind11’s internal data.
The second point is quite a problem as there is more documentation but only with using the Python C-API, not dealing with Pybind objects 1 2 3
I have looked at the documentation but it doesn't address exactly my problem because it doesn't call Python within multithreading. I have also looked at this 4 but the interpreter is called from c++ and not Python
Any hint would be highly appreciated as I have been blocked for quite some time on this problem. Here is a sample code to illustrate the problem.
py::object create_seq(
py::object self
){
std::vector<py::object> dict = self.cast<std::vector<py::object>>();
py::gil_scoped_release release;
#pragma omp parallel for ordered schedule(dynamic)
for(unsigned int i=0; i<dict.size(); i++) {
?? WHAT HAPPENS HERE ??
std::cout << i << std::endl;
#pragma omp ordered
dict[i].attr("attribute") = open.attr("parallel")().cast<int>();
}
py::gil_scoped_acquire acquire;
return self;
}
PYBIND11_MODULE(error, m){
m.doc() = "pybind11 module for iterating over generations";
m.def("create_seq", &create_seq,
"the function which creates a sequence");
}
Python code
import error
class test():
def __init__(self):
self.attribute = None
def func():
return 2
if __name__ == '__main__':
dict = {}
for i in range(50):
dict[i] = test()
pop = error.create_seq(list(dict.values()))
Compiled with
g++ -O3 -Wall -shared -std=c++14 -fopenmp -fPIC `python3 -m pybind11 --includes` openmp.cpp -o error.so
