Python setuptools: How can I list a private repository under install_requires?

Question

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:

from setuptools import setup
setup(name='my_project',
    ...,
    install_requires=[
        'public_package',
        'other_public_package',
        'private_repo_1',
        'private_repo_2',
    ],
    dependency_links=[
        'https://github.com/my_account/private_repo_1/master/tarball/',
        'https://github.com/my_account/private_repo_2/master/tarball/',
    ],
    ...,
)

I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer.

The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate.

Here are some things I've tried:

Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http.
https://<username>:<password>@github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.)
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.

Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)

Duplicate-ish question with no answers here.

score 104 · Accepted Answer · edited Mar 16 '20 at 10:43

104

I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case).

Please note; this is with pip 20.0.2 on Python 3.7.4

setup(
    name='<package>',
...
    install_requires=[
        '<normal_dependency>',
         # Private repository
        '<dependency_name> @ git+ssh://git@github.com/<user>/<repo_name>@<branch>',
         # Public repository
        '<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>',
    ],
)

After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links).

References [1] https://github.com/pypa/pip/issues/4187 [2] https://github.com/pypa/pip/issues/5566

edited Mar 16 '20 at 10:43

Muhammad Usman Bashir

1,441
2
14
43

answered Dec 10 '18 at 12:56

Tom Hemmes

2,000
2
17
23

4

If you use `ssh://` and run into `Could not resolve hostname` change the `:` to `/` in your clone url. I had this error with gitlab. – delijati Apr 15 '19 at 11:53
15

That doesn't seem to work anymore as setuptools seems to look for a package on PyPi with the dependency name: `Reading https://pypi.org/simple/some-fake-name/`, and then `Couldn't find index page for 'some_fake_name' (maybe misspelled?)`. In the end, the last error displayed is `error: Could not find suitable distribution for Requirement.parse('some_fake_name@ git+ssh://git@github.com/cglacet/quadtree.git')` – cglacet Apr 27 '19 at 17:26
2

Emphasis that is not the same as the AND you may replace 'ssh://git@' with 'https://' if it is a public repo – Phil P Aug 14 '19 at 15:35
@Anusha Sorry, I am not able to reproduce cglacet's error. His/her repository is not private and I expect Phil's suggestion to resolve it. Have you tried Phil's suggestion? – Tom Hemmes Feb 20 '20 at 11:18
@TomHemmes I am trying to install a local package, so in `install_requires` I have ` @ file://localhost/lib//.version.whl` and I get the same error as @cglacet – Anusha Feb 21 '20 at 02:03
warning!!!! when you pip upgrade, it seems that even the branch is moving to some new commit, pip regard them as satisfy! I have posted a question here [question](https://stackoverflow.com/questions/65969279/pip-install-upgrade-fail-to-upgrade-private-dependency) – Luk Aron Jan 30 '21 at 15:14
To add a bit more info on @delijati comment, the SSH URL that is given by Gitlab is not interpretable because it has `:` that separates the domain name from the group `com:`. Eg : `git@gitlab..com:/.git` to be replaced by `git@gitlab..com//.git` – Florian Guyot May 18 '22 at 09:42

score 45 · Answer 2 · answered Nov 27 '13 at 05:47

45

Here's what worked for me:

  install_requires=[
      'private_package_name==1.1',
  ],
  dependency_links=[
      'git+ssh://git@github.com/username/private_repo.git#egg=private_package_name-1.1',
  ]

Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.

answered Nov 27 '13 at 05:47

vadimg

592
7
7

3

Hi vadimg - Which version of setuptools/distribute are you using? I get "Unknown url type: git+ssh" using distribute 0.7.3 (the latest version). – Eric P Dec 06 '13 at 21:10
2

This did not work for me 3 years later... Got: Could not find a version that satisfies the requirement – jsmedmar Aug 24 '16 at 16:09
7

As of Oct 28, 2016, this approach seems broken – Max Oct 28 '16 at 17:56
It works for me with `pip==9.0.1` and `setuptools==34.3.1` after I set the same version number in `install_requires` and `dependency_links`. But it seems only work when I run `python setup.py install`, not working on wheels and eggs with pip install. – Kxrr Mar 04 '17 at 14:46
1

With modern pip's you need to include an option, something like `pip install --process-dependency-links ...` – Ash Berlin-Taylor Nov 10 '17 at 17:31
This also works with HTTPS: `git+https://github.com//private_repo.git#egg=private_package-0.0.1` Of course, it will prompt for password (unless it is stored in a keyring). It also requires `--process-dependency-links` option if installing via pip. – RDK Jan 16 '18 at 22:49
2

`--process-dpendency-links` is deprecated, see my answer using PEP508 url specification – Tom Hemmes Dec 10 '18 at 13:02
I have the following error: `unknown url type: git+ssh ` or `unknown url type: git+https` (Python 3.7) – cglacet Apr 27 '19 at 17:14

Maximilian · Answer 3 · 2016-08-31T16:51:41.470

I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.

GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:

from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
version can be a branch, a tag, or a commit hash
You need to supply --process-dependency-links if installing from pip

I am getting a cannot find tag or branch message. Despite the fact that the private repo I am attempting to clone does have a tag. — trendsetter37, Nov 14 '16 at 17:46
figured out what the problem was. The tag had a `v` prepended to it in github. So I needed to use `v1.1.0` instead of `1.1.0` in my setup.py script. — trendsetter37, Nov 14 '16 at 18:25

score 5 · Answer 4 · answered Mar 31 '15 at 19:06

5

I found a (hacky) workaround:

#!/usr/bin/env python

from setuptools import setup
import os

os.system('pip install git+https://github-private.corp.com/user/repo.git@master')

setup( name='original-name'
     , ...
     , install_requires=['repo'] )

I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.

answered Mar 31 '15 at 19:06

cjohnson318

3,154
30
33

yes this was also an ugly workaround for us due to the following: https://github.com/pypa/pip/issues/2822 – Tommy Feb 08 '16 at 20:14
This is the only way I could get it to work, although I went with `import pip`. Neither @vadimg's answer or [this suggestion in pypa/pip](https://github.com/pypa/pip/issues/2124) worked. – gens May 23 '16 at 22:25
5

This will install a dependency even if running something unrelated to installation like `python setup.py --version`. – WhyNotHugo Apr 14 '17 at 20:36

score 4 · Answer 5 · answered Jul 04 '19 at 08:38

4

Via Tom Hemmes' answer I found this is the only thing that worked for me:

    install_requires=[
        '<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']

answered Jul 04 '19 at 08:38

PidgeyBE

71
5

score 1 · Answer 6 · answered Apr 14 '14 at 13:34

1

Using archive URL from github works for me, for public repositories. E.g.

dependency_links = [
  'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]

answered Apr 14 '14 at 13:34

Overclocked

1,187
1
11
22

2

How do you find the `eggname-version`? – cglacet Apr 27 '19 at 17:30

score 1 · Answer 7 · answered Apr 21 '21 at 19:47

1

With pip 20.1.1, this works for me

install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],

in setup.py

answered Apr 21 '21 at 19:47

user15725949

11
1

score 0 · Answer 8 · answered Jun 05 '23 at 10:57

Here is another way (free):

Clone GitHub Repo to GitLab Repo (free account) link
Add GitLab Repo Access Key (read only) [repository -> settings -> access tokens]
Use as dependency: https://oauth2:GITLAB_ACCESS_TOKEN>@gitlab.com/<GITLAB_USERNAME>/<PROJECT_NAME>.git

Q&A:

Q: Why through GitLab? A: Because GitHub only has Account Access Keys, GitLab has Repository Access Keys!

Q: Access Tokens in git history? A: Yes, because the whole point is to use a privat repo in privat context, the token is read-only, and the token expires after maximum of 1 year.

Q: Why is the GitLab Repository not synced automatically with the GitHub Repo? A: Mirroring is a GitLab premium feature, but you can do it for free if you use GitHub Actions, here is a tutorial.

wor · Answer 9 · 2013-09-05T10:04:08.793

-1

Edit: This appears to only work with public github repositories, see comments.

dependency_links=[
    'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
    'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],

Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.

edited Sep 05 '13 at 10:04

answered Sep 01 '13 at 23:30

wor

343
2
6

I still get the same 404 error. Are the repos in your case private? I am aware of the #egg= syntax but don't know that it affects authentication. – Eric P Sep 04 '13 at 22:58
Sorry, it of course was a public repo project. There probably is not any way to do this with current setuptools if https//:@... tarball URLs don't work with githubs private repositories as this question seems also to indicate: [having-trouble-downloading-git-archive-tarballs-from-private-repo](http://stackoverflow.com/questions/10046027/having-trouble-downloading-git-archive-tarballs-from-private-repo) – wor Sep 05 '13 at 09:55
You could try also development version of setuptools and/or check the code. The relevant code seems to be: [package_index:fetch_distribution():534](https://bitbucket.org/pypa/setuptools/src/9f475ec7aedd04ccf89483d8575808d30e31ab51/setuptools/package_index.py?at=default#cl-534) [package_index.py:_download_url():736](https://bitbucket.org/pypa/setuptools/src/9f475ec7aedd04ccf89483d8575808d30e31ab51/setuptools/package_index.py?at=default#cl-736) – wor Sep 05 '13 at 10:50

score -2 · Answer 10 · answered Jan 10 '18 at 18:10

This work for our scenario:

package is on github in a private repo
we want to install it into site-packages (not into ./src with -e)
being able to use pip install -r requirements.txt
being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt

https://github.com/pypa/pip/issues/3610#issuecomment-356687173

Python setuptools: How can I list a private repository under install_requires?

10 Answers10

Linked