0

I am packaging a python project and look for the "right" way to specify dpendencies on external files. I am aware of multiple ways to specify dpendencies, but I would like to know how these ways work practically, how they differ, and at which specific time points they are taking effect, respectively.

Known ways to specify dependencies (see here)

  1. We could specify the dependency in the install_requires argument of setuptools.setup
  2. We could use requirements.txt
  3. We could use setup.cfg

My main questions

  1. What are advantages and disadvantages of the different methods?
  2. When and in which order are the information in the respective files read? Would e.g. pip first execute setup.py and then check the requirements.txt afterwards?
  3. What happens if I specify a requirement only in one of the ways given above? What happens, if I speciefy different requirements with different ways?

Motivating example

I need to create a package that uses cython and numpy. As can be seen e.g. here, cython (and similarly numpy, must be imported before setuptools.setup is called. Hence, setup.py would raise an ImportError if the library is not available. How would I make the requirement visible anyway so that the necessary packages are installed before setup.py is called? Should I move the compilation to a different file that is then called from setup.py? How would I do that?

Samufi
  • 2,465
  • 3
  • 19
  • 43
  • 1
    This probably solves your problem https://stackoverflow.com/q/54117786/5769463 – ead Dec 29 '19 at 04:13
  • @ead Thank you for pointing me to that link. This might solve my particular problem. Nonetheless, I would be interested in understanding the whole process better. Hence, I am looking for answers of the more general questions as well. – Samufi Dec 29 '19 at 14:38
  • 1
    This platform is not really meant to answer such a question which is actually multiple broad questions in one. It's much better to ask multiple focused questions separately instead. `install_requires` and `requirements.txt` serve different purposes, even though there is overlap. For [Cython with setuptools](https://setuptools.readthedocs.io/en/latest/setuptools.html#distributing-extensions-compiled-with-cython). – sinoroc Dec 29 '19 at 15:34
  • @sinoroc I appreciate your effort to keep this platform clean and with well-fitting questions only. I have seen (and benefitted from) similar questions on SO, though. I am interested in a comparison between the requirement specifications. I have not found a source with this information. In your link, e.g., no information is given what a `*.toml` file is and what purposes it suits. I may be able to implement that but woud not have a clue what I am doing. I would like to understand. – Samufi Dec 29 '19 at 15:56
  • 1
    A full answer would be rather long and most likely inevitably too opinionated. We could use the comments to figure out more focused questions together. For example asking clarifications about the cython with setuptools issue is a valid, independent, and interesting question (the answer would be about [PEP518](https://www.python.org/dev/peps/pep-0518/)). `install_requires` vs. `requirements.txt` is also a valid question but already answered multiple times I believe, a good introduction might be this: https://caremad.io/posts/2013/07/setup-vs-requirement/ – sinoroc Dec 29 '19 at 16:28

1 Answers1

1

These are many broad questions. It is difficult to give an exhaustive answer and to keep opinions aside...

When it comes strictly to packaging Python projects, I believe pip's "requirements" files are not commonly useful nor used (there are counter examples of course, but I would argue that they are not a good idea for packaging in general). But in some other use cases, I believe they are very helpful.

If setuptools is used for packaging (there are perfectly valid alternatives), then install_requires is the way to go to declare the direct dependencies of the project. Unless there is no other way around, then I would recommend placing the whole setuptools configuration (this includes install_requires) in the setup.cfg file instead of as arguments to setuptools.setup(). The result would be exactly the same in both cases but (as it is now commonly accepted) for configuration, a static declarative file is a much more natural fit than executable code. As for the order of precedence, I believe a setuptools.setup() argument would override the same configuration item read from setup.cfg.

Would e.g. pip first execute setup.py and then check the requirements.txt afterwards?

These are two different things. The requirements file is more or less a list of projects that one wants to install. Each of these projects might be packaged (setuptools or other) and distributed (source or pre-built wheel) in different ways, some will require the execution of a setup.py some won't. But again, when packaging (and distributing) a project there is usually no need to bother with a requirements file (never when the project is a library, and not at first when the project is an application, see Donald Stufft's article "setup.py vs requirements.txt").

I would rather not go into details regarding the question about the combination of Cython and setuptools. It would be a good additional question. These two links might help get you started though:

sinoroc
  • 18,409
  • 2
  • 39
  • 70