5

I have a really similar problem to Python Multiprocessing Pool Map: AttributeError: Can't pickle local object

I think I understand where the problem is, I am just not sure how to fix it. "Pool.map" needs a top level function as input. But i dont know how i can rewrite this problem:

A simplified code version:

import os as os
from multiprocessing import Pool
import numpy as np
def opti_fun_data(prediction):
    def opti_fun(x):
        def error_fun(i):
            return error_fun_opti(x,prediction,i)
        try:
            pool = Pool(np.max([os.cpu_count()-1,1]))
            error = np.mean(pool.map(error_fun, range(M)))
        finally: # To make sure processes are closed in the end, even if errors happen
            pool.close()
            pool.join()
        return error
    return opti_fun

If I run opti_fun_data(prediction)(x0) i get

Can't pickle local object 'opti_fun_data.<locals>.opti_fun.<locals>.error_fun'

I am new to the multiprocessing library and could use a helping hand. For those who are interested in a bit of background: I want to minimize the function "opti_fun" for a bunch of different scenarios/predictions. Calculating my error measure/benchmark ("error_fun_opti") is quiete computitional intensive, therefore i try to parallelize this step.

Henrik Brautmeier
  • 53
  • 1
  • 5
  • fyi. instead of using a `try-finally` block, you can use a `with` block to ensure the processes close: `with Pool(size) as pool:` – Sintrias Jun 04 '23 at 20:45

1 Answers1

12

Lift the function to the top level and use functools.partial to provide the scoped values rather than nested scope.

def error_fun(x, prediction, i):
    return error_fun_opti(x, prediction, i)

error = np.mean(pool.map(functools.partial(error_fun, x, prediction), range(M)))
Dan D.
  • 73,243
  • 15
  • 104
  • 123