Installing python deps with pip -r requirements.txt fails, but installing one by one succeeds. Why?

Question

I have the following requirements.txt file:

appdirs==1.4.3
decorator==4.3.0
numpy==1.15.4
pybind11==2.2.4
pyopencl==2018.2.2
pytools==2018.5.2
six==1.12.0

Running pip install -r requirements.txt fails with:

Collecting appdirs==1.4.3 (from -r reqs.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting decorator==4.3.0 (from -r reqs.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Collecting numpy==1.15.4 (from -r reqs.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/86/04/bd774106ae0ae1ada68c67efe89f1a16b2aa373cc2db15d974002a9f136d/numpy-1.15.4-cp35-cp35m-manylinux1_x86_64.whl (13.8MB)
    100% |████████████████████████████████| 13.8MB 1.2MB/s 
Collecting pybind11==2.2.4 (from -r reqs.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/f2/7c/e71995e59e108799800cb0fce6c4b4927914d7eada0723dd20bae3b51786/pybind11-2.2.4-py2.py3-none-any.whl (145kB)
    100% |████████████████████████████████| 153kB 1.2MB/s 
Collecting pyopencl==2018.2.2 (from -r reqs.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/bc/58/3ab1246e94986f1b6953e76d7ea7e69d2dbfef7b3f3874eded48524a024f/pyopencl-2018.2.2.tar.gz (341kB)
    100% |████████████████████████████████| 348kB 1.1MB/s 
    Complete output from command python setup.py egg_info:
    ---------------------------------------------------------------------------
    Pybind11 is not installed.
    ---------------------------------------------------------------------------
    Very likely, the build process after this message will fail.

    Simply press Ctrl+C and type
    python -m pip install pybind11
    to fix this. If you don't, the build will continue
    in a few seconds.

    [1] https://pybind11.readthedocs.io/en/stable/
    ---------------------------------------------------------------------------
    Continuing in 1 seconds...    
    ---------------------------------------------------------------------------
    Mako is not installed.
    ---------------------------------------------------------------------------
    That is not a problem, as most of PyOpenCL will be just fine
    without it. Some higher-level parts of pyopencl (such as
    pyopencl.reduction) will not function without the templating engine
    Mako [1] being installed. If you would like this functionality to
    work, you might want to install Mako after you finish
    installing PyOpenCL.

    Simply type
    python -m pip install mako
    either now or after the installation completes to fix this.

    [1] http://www.makotemplates.org/
    ---------------------------------------------------------------------------
    Hit Ctrl-C now if you'd like to think about the situation.
    ---------------------------------------------------------------------------
    Continuing in 1 seconds...   
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 353, in <module>
        main()
      File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 320, in main
        language='c++',
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 41, in __init__
        self._include_dirs = self.include_dirs
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 55, in get_include_dirs
        return self._include_dirs + self.get_additional_include_dirs()
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 52, in get_additional_include_dirs
        return [self.get_numpy_incpath()]
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 47, in get_numpy_incpath
        file, pathname, descr = find_module("numpy")
      File "/home/app/.venv/lib/python3.5/imp.py", line 296, in find_module
        raise ImportError(_ERR_MSG.format(name), name=name)
    ImportError: No module named 'numpy'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-d25rz2_5/pyopencl/

Whereas running:

pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2

works like a charm. Any idea why? Thanks in advance.

--------- EDIT: ---------

I have also tried with pip-tools without success... In this case I created a file requirements.in:

numpy
pybind11
pyopencl

And ran pip-compile, which fails with:

Traceback (most recent call last):
  File "/home/app/.venv/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/scripts/compile.py", line 196, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 101, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 198, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 284, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 217, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 183, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist
    self.req.run_egg_info()
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp0ykgmwk4build/pyopencl/

hoefling · Accepted Answer · 2019-01-08T11:30:52.627

Running pip install -r requirements.txt fails ... Whereas running:
pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2
works like a charm. Any idea why?

That's because the two commands are not identical. pip install supports multiple arguments (installing multiple packages in one pass), so the command equivalent to pip install -r requirements.txt would rather be

$ pip install numpy==1.15.4 pybind11==2.2.4 pyopencl==2018.2.2  # etc

and this one will fail. The reason for this is that pyopencl contains not only Python code, but also C code (so-called "C extension") that needs to be compiled with a C compiler before it can be installed. For the compilation, both numpy and pybind11 must already be installed when pyopencl starts building the extensions, but they are only scheduled for installation at that time. Thus, pip is unable to build pyopencl, thus can't prepare it for installation and in the end, fails the whole command, installing nothing at all.

This is unfortunately quite a common issue. There are two ways to avoid it:

Basically, this is the same approach as the one you've already discovered: install different groups of packages in separate pip install commands. Split the pip install -r requirements.txt command into two: create requirements-pre.txt (name is not relevant). Move numpy and pybind11 from requirements.txt to requirements-pre.txt. Now install the packages with
```
$ pip install -r requirements-pre.txt && pip install -r requirements.txt
```
Use pipenv and Pipfiles instead of pip and "requirements" text files, as pipenv knows how to reinstall failed dependencies in a single pass. Packages persisted in the Pipfile
```
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
numpy = "==1.15.4"
pybind11 = "==2.2.4"
pyopencl = "==2018.2.2"
```

can be installed with a single pipenv install command.

And ran pip-compile, which fails with:

This is the same error as with pip install, only that the original traceback is hidden by pip-compile.

Installing python deps with pip -r requirements.txt fails, but installing one by one succeeds. Why?

1 Answers1