How do I install a Python package from GitHub with extra dependencies, but without `git`?

Question

I am looking for a one-liner to install a Python package from GitHub via pip, without having git installed necessarily, and with being able to specify extra dependencies.

This is what I tried (no need to escape [] as I am on Windows):

pip install https://github.com/python/mypy/archive/master.zip[reports]

The following does work, with the first one taken from this answer:

# requires a git client to be installed
pip install -e git+https://github.com/python/mypy.git#egg=mypy[reports]

# omits extra dependencies
pip install https://github.com/python/mypy/archive/master.zip

But they have various restrictions as noted in the comments.

For the initial command, I am getting

Collecting https://github.com/python/mypy/archive/master.zip[reports]
  ERROR: HTTP error 404 while getting https://github.com/python/mypy/archive/master.zip[reports]
  ERROR: Could not install requirement https://github.com/python/mypy/archive/master.zip[reports] because of HTTP error 404 Client Error: Not Found for url: https://github.com/python/mypy/archive/master.zip%5Breports%5D for URL https://github.com/python/mypy/archive/master.zip[reports]

(I am aware that this is not supposed to work for packages using setuptools_scm, e.g., psf/black, due to https://github.com/pypa/setuptools_scm/blob/ca3855ba66fa4cb100f5039eea909932f815a4a4/src/setuptools_scm/__init__.py#L106-L115, but my package does not depend on that and installs fine from a GitHub zip file.)

I have found the rather obscure solution

pip install "mypy[reports] @ https://github.com/python/mypy/archive/master.zip" 

from How to pip install with wheel specifying extras?

Is there anything more intuitive?

Answer 1

I know you are asking for a one-liner, but let me describe first the common workaround to targeting Git repositories (not just Github) that also avoids requiring git:

1. Download a copy of the source code
   • Using wget or curl or whichever shell or script-able utility is available
   • From Github, you can target either
     • A .zip/.tar.gz of a specific branch or commit or tag
       For example, from mypy's Github, you can get the latest master, a specific tag v0.950, or choose any of the other release tags.
     • A .zip/.tar.gz from a release
       For example, from black's Github, from the 22.3.0 release page, get the 22.3.0 source codes
2. Unzip/Untar the .zip/.tar.gz to some temporary directory
3. From that temporary directory, run

   pip install ./<temp directory>[<extra dependency]
   

   • The same considerations for where pip installs the package applies
   • If using a virtual env, make sure to activate it first
   • Or use the python -m pip install ... syntax to select a specific environment and interpreter

With regular pip install targeting a Git repo like this,

pip install git+https://github.com/python/mypy.git#egg=mypy[reports]
pip install "mypy[reports] @ https://github.com/python/mypy/archive/master.zip"

...it automatically already does steps 1 and 2. If you add a -v/--verbose option, you'll see that it clones a copy of the codes first from a specific commit, and stores them in some temporary directory (which we don't normally care about), then does the regular installation from there.

Sample for mypy:

$ pip install -v git+https://github.com/python/mypy.git#egg=mypy[reports]
...
Collecting mypy[reports]
  Cloning https://github.com/python/mypy.git to /private/var/folders/3h/pdjwtnlx4p13chnw21rvwbtw0000gp/T/pip-install-d6gv_iva/mypy_fe678fe853cb45b68755c0c57dfcb757
...
Cloning into '/private/var/folders/3h/pdjwtnlx4p13chnw21rvwbtw0000gp/T/pip-install-d6gv_iva/mypy_fe678fe853cb45b68755c0c57dfcb757'
...
Building wheels for collected packages: mypy
  Running command Building wheel for mypy (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  pin_version()
  creating build
  creating build/lib
  creating build/lib/mypy
  creating build/lib/mypyc
...
Successfully built mypy
Installing collected packages: mypy
  Running command git rev-parse HEAD
  4f07c79aea0fef61ab649d6acedf01186f1054eb
  changing mode of /path/to/venv/bin/dmypy to 755
  changing mode of /path/to/venv/bin/mypy to 755
  changing mode of /path/to/venv/bin/mypyc to 755
  changing mode of /path/to/venv/bin/stubgen to 755
  changing mode of /path/to/venvi/bin/stubtest to 755
Successfully installed mypy-0.960+dev.4f07c79aea0fef61ab649d6acedf01186f1054eb

$ python
...
>>> import importlib_metadata
>>> importlib_metadata.metadata('mypy').get_all('Requires-Dist')
[ ... extra == 'reports'"]

So, if you don't have Git, you can't do a git clone, but instead have to download the codes manually. As with the direct pip install, if you don't care about a specific version, you can just download the latest master/main commit:

Sample again for mypy:

$ wget -O mypy-master.zip https://github.com/python/mypy/archive/refs/heads/master.zip
...
2022-05-05 20:29:07 (2.85 MB/s) - ‘mypy-master.zip’ saved [3433301]

$ unzip -o mypy-master.zip
...
inflating: mypy-master/test-data/unit/typexport-basic.test  
inflating: mypy-master/test-requirements.txt  
inflating: mypy-master/tox.ini 

Then the last step would simply be doing pip install, which supports targetting a <local project path> and specifying extras all in one command:

python -m pip install .[PDF]  # project in current directory

So following the downloaded and unpacked mypy-master above, the last step would simply be:

$ pip install -v ./mypy-master[reports]
Processing ./mypy-master
...
Building wheels for collected packages: mypy
  Running command Building wheel for mypy (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  pin_version()
  creating build
  creating build/lib
  creating build/lib/mypy
  creating build/lib/mypyc
...
Successfully built mypy
Installing collected packages: mypy
  Running command git rev-parse HEAD
  4f07c79aea0fef61ab649d6acedf01186f1054eb
  changing mode of /path/to/venv/bin/dmypy to 755
  changing mode of /path/to/venv/bin/mypy to 755
  changing mode of /path/to/venv/bin/mypyc to 755
  changing mode of /path/to/venv/bin/stubgen to 755
  changing mode of /path/to/venvi/bin/stubtest to 755
Successfully installed mypy-0.960+dev.4f07c79aea0fef61ab649d6acedf01186f1054eb

$ python
...
>>> import importlib_metadata
>>> importlib_metadata.metadata('mypy').get_all('Requires-Dist')
[ ... extra == 'reports'"]

...which shows the same output as the one that pip install-ed directly from git+https. The same considerations with using pip install applies, for example, if you are using a virtual environment, make sure to activate that first so mypy is installed in the same location.

Now, to make it all a one-liner command, you can turn those steps into a script, a shell alias, or just straight-up run multiple commands with && operator:

$ wget -O mypy-master.zip https://github.com/python/mypy/archive/refs/heads/master.zip && unzip -o mypy-master.zip && pip install ./mypy-master[reports] && rm -R mypy-master*

Note that I added removal of the temporary files/folders at the end of the command. If one of the steps fails (ex. network connection error), then the rest of the commands will be aborted. You can add better/more error-handling as needed.

$ wget -O mypy-master.zip https://github.com/python/mypy/archive/refs/heads/master.zip && unzip -o mypy-master.zip && pip install ./mypy-master[reports] && rm -Rf mypy-master*
--2022-05-05 21:09:39--  https://github.com/python/mypy/archive/refs/heads/master.zip
Resolving github.com (github.com)... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ‘github.com’

I agree though that that isn't an elegant or an intuitive command.