81

I have dependency_links in my setup.py: 

...
dependency_links = ['http://github.com/robot-republic/python-s3/tarball/master.tar.gz#egg=python-s3'],
...

But it doesn't work. However install_requires works fine. Maybe there are another method to set up git repo as required for setup.py?

syabro
  • 1,857
  • 2
  • 15
  • 30
  • 9
    It seems that everyone is saying `dependency_links` is deprecated, and you're supposed to use requirements.txt instead. But this is no good if you want your dependencies automatically installed when you pip-install your repo. Does anyone know what you're supposed to do now? pip are you there? – Peter Jun 16 '17 at 07:25

6 Answers6

69

This answer should help. In a nutshell, you need to specify the version (or "dev") for the #egg=python-s3 so it looks like #egg=python-s3-1.0.0.

Updates based on @Cerin's comment:

  • Pip 1.5.x has a flag to enable dependency-links processing: --process-dependency-links. I haven't tested it because I agree with the point below.
  • This discussion seems to indicate that using dependency-links for pip is a bad practice. Although this feature was enlisted for deprecation, it's not anymore. There's a valid use case for private packages.
OrangeDog
  • 36,653
  • 12
  • 122
  • 207
Laur Ivan
  • 4,117
  • 3
  • 38
  • 62
  • 1. that answer refers to easy_install, not pip, 2. this still doesn't work. For me, pip reports success but nothing is installed to site-packages – Cerin Jan 24 '14 at 17:19
  • 4
    The `--process-dependency-links` flag worked for me. In my case I also had to specify the name of a version that wasn't on pypi. If same version was on pypi and declared in the egg name in `dependency_links`, pip would ignore the `dependency_links` version and just fetch from pypi. This is with pip 6.0.8 on python 3.3.6. – turtlemonvh Apr 01 '15 at 18:01
  • 4
    What is the alternative to using --process-dependency-links ? I am trying to install a repo using `pip` that depends on a github repo. – Ankur Agarwal Sep 22 '15 at 05:57
  • 6
    Looking at https://github.com/pypa/pip/pull/1955 the dependency_links option will survive because it is needed for private builds. – Doncho Gunchev Nov 27 '15 at 12:40
  • I can confirm that pip ignores the entries in `dependency_links` if they do not end with `-maj.min.rev` (pip v1.5.4). You need the version. I am working with private pkgs in git submodules, and I don't want to specify the version inside my requires. Turns out that pip will succeed to resolve my custom dependency link (a `file://path/to/submodule-X.Y.Z` URL) regardless of if the submodule's package matches X.Y.Z. I just put "-0.0.0". And this is with no version specified in the `requires` field. – init_js Apr 06 '18 at 06:39
  • 32
    As of pip version 18.1 (at least) use PEP 508 direct reference specifications. Do not use dependency_links. Example: `install_requires=['python-s3 @ http://github.com/robot-republic/python-s3/tarball/master.tar.gz']` – Wolfgang Kuehn Dec 27 '18 at 18:14
  • 10
    @WolfgangKuehn you should consider converting your comment into an actual answer. It's the only thing that helped me out of all the SO posts on this topic, and it's burried in the comments list. – andreimarinescu Aug 14 '19 at 14:05
61

since pip version 18.1 PEP 508 URL is supported. That means you don't need the deprecated dependency_links anymore. You write the dependency directly in the install_requires list instead. The example from @Chad looks like this:

setup(
    name='yourpackage',
    version='1.7.5',
    packages=[],
    url='',
    license='',
    author='',
    author_email='',
    description='',
    install_requires=[
        'somepackage==1.2.0',
        'repo @ https://github.com/user/archive/master.zip#egg=repo-1.0.0',
        'anotherpackage==4.2.1'
    ],
)

To install your package you can simply write:

pip install yourpackage

(without --process-dependency-links)

Anentropic
  • 32,188
  • 12
  • 99
  • 147
Easy_Israel
  • 841
  • 6
  • 8
  • 4
    I get error from pip 19.0.3 `Invalid requirement, parse error at "'@ https:'"` – Jaakko Mar 19 '19 at 10:54
  • 11
    I had to remove the version specifier. I changed `'repo==1.0.0 @ https://github.com/user/archive/master.zip#egg=repo-1.0.0'` to `'repo @ https://github.com/user/archive/master.zip#egg=repo-1.0.0'` to make it work. – Jaakko Mar 19 '19 at 11:44
  • 2
    @Jaakko youre right, removing the `==1.0.0` works great. But the issue remains, Im currently trying to override a package that is currently in pypi, when i do: `validators@https://github.com/kingbuzzman/validators.git#egg=validators-0.13.0` i get the following output: https://gist.github.com/kingbuzzman/e3f39ba217e2c14a9065fb14a502b63d and it's finding 0.12.6 and using that instead. NOT what i was expecting to happen. – Javier Buzzi May 08 '19 at 16:47
  • 1
    Follow up, looks like if you use `pip` to install, everything works as expected. When you use the native `python setup.py install` or `python setup.py develop` it installs the wrong version every time. Discussion can be followed here: https://stackoverflow.com/questions/56046146/setup-py-ignores-full-path-dependencies-instead-looks-for-best-match-in-pypi – Javier Buzzi May 08 '19 at 17:33
  • 2
    These pep508 urls only work for source dists, not for wheels. Is that by design, or should this considered to be a bug? (I am using setuptools 39.0.1) – jjmurre Jun 27 '19 at 06:59
  • Ok, should have checked this earlier, pep508 urls are working for wheels when using setuptools==41.0.1 – jjmurre Jun 27 '19 at 08:21
  • 1
    PEP508 says that either the version specifier or the URL is allowed, not both: ``specification = wsp* ( url_req | name_req ) wsp*`` – VPfB Mar 08 '20 at 15:09
47

I realize this is an old question, but, just in case you find yourself here like I did, this is what worked for me.

I've got a package on GitHub (not registered with pypi) that relies on other GitHub (non-pypi) packages. I spent an inordinate amount of time trying to figure out how to get pip to handle this correctly. I will include what I did to fix it here.

Putting dependencies in a requirements.txt file is the preferred method of listing dependencies. However, you also need to populate install_requires in setup. It was at this stage that I ran into a roadblock with pip not wanting to install dependencies from GitHub.

Most places, including answers to this question, tell you to populate the dependency_links section of setup. However, you also need to populate the install_requires field with the name of the package referenced in dependency_links.

For example, if your requirements.txt contains the following.

somepackage==1.2.0
https://github.com/user/repo/tarball/master#egg=repo-1.0.0
anotherpackage==4.2.1

Then, your setup call should look like this:

setup(
    name='yourpackage',
    version='1.7.5',
    packages=[],
    url='',
    license='',
    author='',
    author_email='',
    description='',
    install_requires=[
        'somepackage==1.2.0',
        'repo==1.0.0',
        'anotherpackage==4.2.1'
    ],
    dependency_links=[
        'https://github.com/user/repo/tarball/master#egg=repo-1.0.0'
    ]
)

Ok, so now we've got our package configured; installing it is the next task. This is where I spent a lot of time. I could not figure out why specifying dependency_links apparently did nothing. The trick is that in some cases, you need to set the allow-all-external (can be more specific) flag for pip. For example:

pip install git+https://github.com/user/anotherrepo.git
--process-dependency-links --allow-all-external

You're done and it works!

DISCLAIMER: dependency_links and the flags process-dependency-links and allow-all-external are deprecated, so they will be removed soon. In the time I spent, I could not locate a better, prefered method and still have pip function properly.

ndmeiri
  • 4,979
  • 12
  • 37
  • 45
Chad
  • 499
  • 4
  • 6
  • 22
    I added this because it seemed a more complete example. Not too many places mention that what is in the dependency links section must also be in the install_requires section. – Chad Jul 18 '16 at 04:01
  • 1
    Thanks @chad ! The trick is to add a version to the dependency + add it also to requirements list. – mvpasarel Nov 22 '17 at 11:58
  • One thing to note, is that the version number is essential when doing this (whereas you can usually omit in in requirements.txt) – cedd Oct 12 '18 at 15:06
  • As of pip version 10.0.0b1 (2018-03-31), `--allow-external` is deprecated. – Wolfgang Kuehn Dec 27 '18 at 17:49
  • 1
    re: preferred method. From https://github.com/pypa/pip/issues/4187#issuecomment-415067034 : `Dependency links are a setuptools-specific mechanism, and are processed by setuptools' internal machinery, not by pip. So unlike direct URL links, we don't have any control over what they do. That's why we deprecated them in favour of the standard direct URL form, which we do handle ourselves.` I believe that "direct URL links" means that you use the full url (including the `git+https` and `#egg=` bits) as an argument to pip. – cowlinator Oct 29 '19 at 01:44
14

A couple of notes on some issues I found, in particular for installing from private repos.

Installing from pip & setuptools have some subtle differences; but this way should work for both.

from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

  • For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
  • You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
  • You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
  • version can be a branch, a tag, or a commit hash
  • You need to supply --process-dependency-links if installing from pip
Maximilian
  • 7,512
  • 3
  • 50
  • 63
10

First upgrade your pip version as this is a new syntax.

pip install pip --upgrade

Then do the following:

install_requires=[
    'bleualign-git @ https://github.com/rsennrich/Bleualign/archive/<commit-hash or branch-name>.zip#egg=bleualign-git-1.0.0'
]
  • Version 1.0.0 and name "bluealign-git" are randomly chosen.
  • The version number is necessary.
  • We advisedly used the name "bleualign-git" to distinguish it from the main repository version.

Hope this helps.

Comments:

Good answer (upvoted). Also want to mention that the actual result would likely depend on the platform and/or pip version, though. I've seen when it works, or seemingly works but did not really pull the dependency from the specified link, or simply being rejected. So I would probably just use this as a short time workaround only. Stick with the mainstream way whenever possible.

RayLuo
  • 17,257
  • 6
  • 88
  • 73
Tobias Ernst
  • 4,214
  • 1
  • 32
  • 30
0

None of the above solutions worked for me in their respective explicit forms. I am adding this as another solution for certain specific cases. OP has a tarball in their remote repo, but I ended up here for a related case: only source code in the remote repo.

I have a dependency that is a remote private repository on Azure DevOps. I would like to use pip to install a local python package and also automatically install the remote dependency. The remote repository is a python package with a setup.py and some source code. The repo doesn't have build artifacts like eggs/wheels/tarballs, just .py files. I am able to manually install the remote dependency with pip install git+https://..., hence I suspected correctly that the remote repository doesn't need to contain eggs/wheels/tarballs.

Here is the relevant line of my setup.py file

setup.py

...

install_requires=["packagename @ git+https://dev.azure.com/.../_git/packagename"]
Julian Irwin
  • 172
  • 10