Scikit-learn machine learning models training using multiple CPUs

Question

I want to decrease training time of my models by using a high end EC2 instance. So I tried c5.18xlarge instance with 2 CPUs and run a few models with parameter n_jobs=-1 but I noticed that only one CPU was utilized:

Can I somehow make Scikit-learn to use all CPUs?

MWE... all that jazz... you should know better given your rep. That being said, as a first pass, I would [make sure that you have compiled numpy with a linear algebra library that actually supports multi-core processing.](https://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-scipy) — Paul Brodersen, Sep 04 '18 at 15:38
@PaulBrodersen I have `numpy` linked to `BLAS`, I hope you understood that it's not about multi-core processing but about multi-cpu processing, my first CPU does the job perfectly fine but the second one is idle. — Michael Dz, Sep 04 '18 at 19:49
Try adding `import multiprocessing; multiprocessing.set_start_method('forkserver');` at the top of your code, before running or importing anything.. Does it solve the issue? — Qusai Alothman, Sep 09 '18 at 00:24
@QusaiAlothman It worked! Could you post the advice as the answer so I can mark it as solved? — Michael Dz, Sep 22 '18 at 20:12

score 1 · Accepted Answer · answered Sep 24 '18 at 13:29

1

Try adding:

import multiprocessing
multiprocessing.set_start_method('forkserver')

at the top of your code, before running or importing anything. That's a well-known issue with multiprocessing in python.

answered Sep 24 '18 at 13:29

Qusai Alothman

1,982
9
23

Scikit-learn machine learning models training using multiple CPUs

1 Answers1