How to create Python requirements.text file without sub-dependencies?

Question

Im quit new to python coming from the JS world .

When installing a package in javascript all the subdependencies of that packge are not part of the package.json. dependencies section. For example:

npm init -y
npm install express

would produce

  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2"
  }

even though express.js has 31 dependencies

Reading about python dependencies. I did the following steps:

created virtual env:

 python -m venv my-virutal-env

made sure no package is installed:

 pip freeze > to-uninstall.txt
 pip uninstall -r to-uninstall.txt
 pip freeze > requirements.txt

Installed llamaIndex

 pip install llama_index

producing requirements again using pip freeze > requirements.txt

produces:

...

llama-index==0.5.12
...

when ... are a lot of other packages

This is very undesirable. How Can I overcome this?

"This is very undesirable." How so? The entire point of requirements.txt is reproducibility, which needs all dependencies. — MisterMiyagi, Apr 10 '23 at 14:20
`pip freeze` lists every third-party package installed in the current environment. All those other packages were installed when you ran `pip install llama_index`. You can manually edit `requirements.txt` to just have the `llama-index==0.5.12` line, and it'll install just fine. However, as MisterMiyagi said, you want to make sure all dependencies and their versions are listed for reproducibility. — MattDMo, Apr 10 '23 at 14:20
Although pip is still the most common way to install packages for python, its not the most feature rich. You might wish to check out python-poetry which may have the kind of features you are after, it seems similar to me to npm and package.json in js. — Andrew Allaire, Apr 10 '23 at 14:26
@MattDMo MisterMiyagi thanks for the responses. I don't think I want to be handling the dependencies of my dependencies. The whole point of a package is that it is like a black box, and this black box should manage its dependencies. This is at least my approach to it. Let's say llama_index discovered a bug and while fixing it upgraded a version of one of their dependencies. if they do it without creating a new version my code will break. About the solution suggested, this is very unscalable. Every time I will install a new package I will need to delete all sub dependencies — yoty66, Apr 10 '23 at 14:30
The [pipreqs](https://pypi.org/project/pipreqs/) library can be used to auto-generate the `requirements.txt` file - using *only* your project’s dependencies. — S3DEV, Apr 10 '23 at 14:33
*I don't think I want to be handling the dependencies of my dependencies* But you do. By providing a complete `requirements.txt`, you are saying to the end user that your package works in *exactly* this environment. It may work in others, it may not, but I know it works here. The user is of course free to update any of the [sub]dependencies as they wish, but if something breaks it's on them. *The whole point of a package is that it is like a black box* I disagree with that statement in several ways, but even if it is, you are programming that black box. You're responsible for the dependencies. — MattDMo, Apr 10 '23 at 14:42
@MattDMo But it doesn't mean that. As long as you are not bundling the dependencies to the source code you are fetching it from some registry. So the source of truth for the sub-dependency is still that remote registry. Creating a ``` requirements .text``` is like creating a "snapshot" of something dynamic, and hence it is breakable in cases like the example I stated in my previous comment. This is why I don't see any reasoning to do so unless you have control over your dependency source code (by bundling it or any other way ) — yoty66, Apr 10 '23 at 16:02
`requirements.txt` isn't there to define the *dependencies* of your *package*. That is the job of the package distribution, commonly via `pyproject.toml`. `requirements.txt` is there to define the *Python environment* of a *program* for reproducibility. — MisterMiyagi, Apr 10 '23 at 16:23
"Creating a ``` requirements .text``` is like creating a "snapshot" of something dynamic, and hence it is breakable in cases like the example I stated in my previous comment." The Python package index is *immutable*; it only accepts additions, not modifications. There is no way for a package to change its code or dependencies "without creating a new version". That ties into the entire point of `requirements.txt`: specifying a set of environment package versions known to be good. — MisterMiyagi, Apr 10 '23 at 16:27
Does this answer your question? [requirements.txt vs setup.py](https://stackoverflow.com/questions/43658870/requirements-txt-vs-setup-py) — MisterMiyagi, Apr 10 '23 at 16:28

jsbueno · Answer 1 · 2023-04-10T19:40:50.603

Using the pattern pip freeze >requirements.txt is just a suggestion, mostly derived from documentation that followed a line of "make your code work first, with everything you need. Now so that others can run your code, use a dump of everything that is installed as your requirements file"

For a small number of known dependencies, just edit your requirements.txt file by hand. You can either edit it from scratch, or, if desired, simply remove all lines for projects you know will be brought along automatically when the higher-level, explicitly needed dependency is installed. Also, note that the sucessor of requirements.txt, the pyproject.toml file, used for more recent tooling such as poetry, works on this basis - but will also need you to add manually the known explicit dependencies of your project. The only difference for your situation is that (Poetry in particular, not sure about other tools), updates the pyproject.toml tool itself as you go adding the explicit dependencies, while as in requirements.txt you are supposed to edit that file.

How to create Python requirements.text file without sub-dependencies?

1 Answers1