12

I have a lot of packages to install in my pip requirement and I'd like to process them in parallell.

I know that, for example, that if I want n parallel jobs from make I have to write make -j n; is there an equivalent command for pip requirements?

Thanks!

Rowandish
  • 2,655
  • 3
  • 31
  • 52

2 Answers2

18

Sometimes pip uses make to build dependencies. If before it starts you set MAKEFLAGS like:

export MAKEFLAGS="-j$(nproc)"
pip install -r requirements.txt

This may help building native dependencies.

Note: nproc resovles as the number of CPUs in your system.

Eddy del Valle
  • 197
  • 1
  • 3
-4

I think that the best approach to have a better speed is to see where is the bottleneck. Try to analyse which processes are taking place when you use pip command.

Probably the most time is spent downloading from PyPI and for compiling the libraries to native (such as PIL). You could try to make your own PyPI repository and to pre-compile the sources that are needed to be compiled. In the past there has been much talk of this but there isn't a really speed up if launching pip in parallel.

Brian Minton
  • 3,377
  • 3
  • 35
  • 41
Maksim
  • 215
  • 2
  • 12
  • 2
    Ok, then just install that package once and then go to site_packages and copy it when you need it. OS and architecture must need to match in order to achieve it and then invoke pip install so it could find that is already installed. – Maksim Oct 31 '14 at 11:05