14

I have a package which I'm pushing to PyPi and some of the depedencies are not packages, but installable git repositories. My requirements.txt looks like this

sphinx_bootstrap_theme>=0.6.5
matplotlib>=2.2.0
numpy>=1.15.0
sphinx>=1.7.5
sphinx-argparse>=0.2.2
tensorboardX
tqdm>=4.24.0
Cython>=0.28.5

# git repos
git+git://github.com/themightyoarfish/svcca-gpu.git

Accordingly, my setup.py has this content:

#!/usr/bin/env python

from distutils.core import setup
import setuptools
import os

with open('requirements.txt', mode='r') as f:
    requirements = f.read()
    required_pkgs, required_repos = requirements.split('# git repos')
    required_pkgs = required_pkgs.split()
    required_repos = required_repos.split()

with open('README.md') as f:
    readme = f.read()

setup(name=...
      ...
      packages=setuptools.find_packages('.', include=[...]),
      install_requires=required_pkgs,
      dependency_links=required_repos,
      zip_safe=False,   # don't install egg, but source
)

But running pip install <package> does not actually install the git dependency. I assume that pip doesn't actually use the setup script. It works when I run python setup.py install manually.

Edit:

I also tried removing dependency_links and just using install_requires with the repository, but when installing my repository from GitHub (the project including the above files), I'm met with

    Complete output from command python setup.py egg_info:
error in ikkuna setup command: 'install_requires' must be a string or 
list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+git://g'"

It has been suggested in other answers that one can put something like

git+https://github.com/themightyoarfish/svcca-gpu.git#egg=svcca

into requirements.txt, but that fails with

   error in <pkg> setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'

Question: (How) Can I list git repositories as dependencies for a pip package?

oarfish
  • 4,116
  • 4
  • 37
  • 66
  • Possible duplicate of [How to state in requirements.txt a direct github source](https://stackoverflow.com/questions/16584552/how-to-state-in-requirements-txt-a-direct-github-source) – Bram Vanroy Feb 26 '19 at 15:10
  • 1
    The linked question and top answers do not seem to solve the problem because it deals with requirements files, but not with `pip`. The idea is having to state the requirement in a way that `setup()` understands it. – oarfish Feb 26 '19 at 21:33
  • 1
    I should have made it clearer in the question that `requirements.txt` is merely a proxy for `setuptools` dependencies in this case. – oarfish Feb 26 '19 at 21:44

3 Answers3

14

Out of the 50 or so different ways to specify git dependencies for Pip, the only one that did what I intended was this one (outline in PEP 508):

svcca @ git+ssh://git@github.com/themightyoarfish/svcca-gpu

This can be used in install_requires, which solves the issue of dependency_links being ignored by pip.

An amusing side-effect is that the package cannot be uploaded to PyPi with such a dependency:

HTTPError: 400 Client Error: Invalid value for requires_dist. Error: Can't have direct dependency: 'svcca @ git+ssh://git@github.com/themightyoarfish/svcca-gpu' for url: https://upload.pypi.org/legacy/
oarfish
  • 4,116
  • 4
  • 37
  • 66
  • 1
    I am stuck on " Error: Can't have direct dependency:" issue and not sure how to solve it :-( – Thamme Gowda Mar 30 '21 at 03:40
  • 3
    "Don't use PyPi" is the only solution until they change this. – oarfish Mar 30 '21 at 08:20
  • 3
    I *had to* release a package to PyPi. So I ended up releasing two packages: first, the direct dependency, and then releasing my actual package after converting direct dependency to regular dependency! – Thamme Gowda Mar 30 '21 at 18:49
  • Ran into the same problem with pypdfium2 ([issue 177](https://github.com/pypdfium2-team/pypdfium2/issues/177)). Not so amusing after all. – mara004 Feb 22 '23 at 21:13
1

We tried it with pyproject.toml and it didn't work for us either.

requires = [
    "geci_plots @ git+https://github.com/IslasGECI/geci_plots.git@v0.4.0",
    "lmfit",
    "pandasql",
    "scikit-learn==1.1.3",
]

It seems that direct references don't allow it.

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Invalid value for requires_dist. Error: Can't have direct dependency:  
         'geci_plots @ git+https://github.com/IslasGECI/geci_plots.git@v0.4.0'

I assume the direct references are the GitHub repos.

user12520082
  • 11
  • 2
0

According to the next post related to How to state in requirements.txt a direct github source. You could add a package from git remote repository with the next syntax:

-e git://github.com/themightyoarfish/svcca-gpu.git

Reference: Install a project in editable mode (i.e. setuptools “develop mode”) from a local project path or a VCS url with-e

R. García
  • 815
  • 9
  • 20
  • I don't want to use editable mode, but it doesn't seem to work without `-e`. – oarfish Feb 26 '19 at 14:27
  • See edited question; this does not work because `install_requires' must be a string or list of strings containing valid project/version requirement specifiers;`, which apparently that is not. – oarfish Feb 26 '19 at 21:32
  • Hi oarfish! I really recommend you follow the next structure created by kennethreitz. On this repository you will see how to create an awesome [Setup.py](https://github.com/kennethreitz/setup.py) – R. García Feb 26 '19 at 21:35
  • Maybe, you should split a break space, something like that: `required_pkgs = required_pkgs.splitlines()` [Reference](https://www.reddit.com/r/Python/comments/3uzl2a/setuppy_requirementstxt_or_a_combination/) – R. García Feb 26 '19 at 21:42
  • I really don't see how your linked `setup.py` file deals with non-pypi packages at all. – oarfish Feb 26 '19 at 21:46
  • No, it was just in order to improve and standarize your setup.py file. – R. García Feb 26 '19 at 21:48