13

I want to add a package to my requirements.txt that would correspond to the command line

pip3 install foo --index-url https://foo.com/bar/baz

I appended 

--index-url https://foo.com/bar/baz
foo

to the end of requirements.txt in the hope that the index-url option will affect only things that come after it. It seems to work (at least as long as I am on the VPN from which foo.com is visible).

Is this the right approach? Thanks.

sds
  • 58,617
  • 29
  • 161
  • 278
  • The `pip install -r requirements.txt` is effectively the same as doing `cat requirements.txt | xargs pip install`, so in your case the custom index arg will be applied to all packages. – hoefling Jul 16 '19 at 08:01
  • Probably more like `grep -v '^#' requirements.txt | xargs pip install` but I see your point. Thanks. Do you know how people handle the issue of requirements coming from different sources? – sds Jul 16 '19 at 13:43
  • 1
    I'd say it depends on your use case. If your use case is _download selection of dependencies from private index, rest from PyPI_ because private deps are not uploaded to PyPI, replace `--index-url` with `--extra-index-url` so `pip` first tries to download from PyPI and resorts to private index on failures. – hoefling Jul 16 '19 at 13:56
  • 1
    If your use case is e.g. _prevent pip from downloading packages from PyPI to protect against package spoofing_, then you won't be able to solve it locally via flags and need to configure the private index accordingly. I would e.g. recommend `devpi` as it has package spoofing prevention turned on by default and point `--index-url` to your private index, so it can decide itself whether to proxy the download requests to PyPI. – hoefling Jul 16 '19 at 13:59

1 Answers1

16

The arguments in requirements.txt are applied to all packages; the command

$ pip install -r requirements.txt

with requirements.txt being

foo
bar>1
baz==2
--flag

is effectively the same as running

$ pip install "foo" "bar>1" "baz==2" --flag

If you want to download only a selection of dependencies from your private index, use --extra-index-url instead of --index-url. This will instruct pip to download packages from PyPI if available, and resort to your private index otherwise (multiple --extra-index-urls are supported, too).

To handle the vice versa - download from private index if available, fallback to PyPI - set your private index as primary, PyPI as extra index:

--index-url=https://my.index/ --extra-index-url=https://pypi.org/simple

If you have other use cases, for example protection against package spoofing, this can't be effectively solved with pip. There are, however, index servers like devpi that can proxy download requests to PyPI and offer spoofing protection out of the box.

Edit: @Geordie explained package spoofing in his comment well.

hoefling
  • 59,418
  • 12
  • 147
  • 194
  • Looks like you can also use the --no-index flag for even more control: --no-index --find-links /my/local/archives --find-links http://some.archives.com/archives – Geordie Dec 12 '19 at 20:34
  • 4
    Also be aware that pip still tries to source the latest version of the module if you don't specify it in requirements.txt. So even though you set your primary index to myIndex and extra to pypi, if pypi contains a higher version of the same library than myIndex it will still pull the package from pypi – Geordie Dec 12 '19 at 20:56
  • Although it worked for me to get installed de dependencies applying `pip install -r requirements.txt` but still I was getting error when using _requirements.txt_ for declaring the dependencies of my package built by `setup.py`. Finally, I solved this issue introducing the corresponding url as `setup( ... dependency_links=[ "http://some.archives.com/archives" ], ...)` as described [here](https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html) – Elias Oct 13 '20 at 14:22