0

I am using dask on LSF cluster to process some images in parallel. The processing function itself uses joblib to perform multiple computations on the image in parallel.

It seems that setting n_workers and cores parameters to some numbers will generally produce n_workers * cores futures running at the same time. I would like to have n_workers futures being processesed at a time, each of them having cores cores at disposal for the purpose of using them with joblib.

How do I achieve such result?

Smorts
  • 95
  • 6
  • Does this answer your question? [Dask - how to assign task to the specific CPU](https://stackoverflow.com/questions/66162626/dask-how-to-assign-task-to-the-specific-cpu) – SultanOrazbayev Jul 20 '22 at 06:21
  • 1
    @SultanOrazbayev I am not sure, maybe? Do you suggest that I could solve the problem by setting each Worker to have available resource of 1 CPU and then each task to also require 1 CPU? I can only test it tommorow. Is my understanding right? – Smorts Jul 20 '22 at 07:39
  • 1
    Yes, if I understand the situation correctly, that should resolve it. – SultanOrazbayev Jul 20 '22 at 08:18

0 Answers0