poetry dependencies not available when running script with tox

Question

I have a python project that uses poetry and tox. It has source code, tests and scripts (juptext notebooks). I can't import the dev dependencies in the scripts, but I can in the tests.

When I came across this problem, I created the following minimal example. At first, it didn't work, then I fiddled around with it, and now it's working. So I stripped the project that has the actual problem down so it's indistinguishable other than the project name, location, virtual env, and .git directory, but that's still not working.

UPDATE deleting all build artifacts and the virtualenv for the minimal example makes it stop working again

UPDATE adding the line scripts: poetry install to the tox commands fixed only the minimal example

The source code, tests and scripts are in the following layout

foo
  +--foo
  |  +--__init__.py
  |
  +--tests
  |  +--__init__.py
  |  +--test_foo.py
  |
  +--scripts
  |  +--foo_script.py
  |
  +--pyproject.toml
  +--tox.ini

The files are either empty or as follows:

foo_script.py

import requests

test_foo.py

import requests
import pytest

def test():
    assert True

pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]

[tool.poetry.dependencies]
python = "^3.7"
requests = "*"

[tool.poetry.dev-dependencies]
pytest = "^4.6"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

tox.ini

[tox]
envlist = test, scripts
isolated_build = true
skipsdist = true

[testenv]
basepython = python3.7
whitelist_externals =
    pytest
    bash
commands =
    test: pytest
    scripts: bash -c 'python3 scripts/*.py'

When I run tox, I get

test run-test-pre: PYTHONHASHSEED='4126239415'
test run-test: commands[0] | pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
cachedir: .tox/test/.pytest_cache
rootdir: /home/#######/foo
collected 1 item                                                               

tests/test_foo.py .                                                      [100%]

============================== 1 passed in 0.09s ===============================
scripts run-test-pre: PYTHONHASHSEED='4126239415'
scripts run-test: commands[0] | bash -c 'python3 scripts/*.py'
Traceback (most recent call last):
  File "scripts/foo_script.py", line 1, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
ERROR: InvocationError for command /bin/bash -c 'python3 scripts/*.py' (exited with code 1)
___________________________________ summary ____________________________________
  test: commands succeeded
ERROR:   scripts: commands failed

Answer 1

I believe something like the following should work:

pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]

[tool.poetry.dependencies]
python = "^3.7"
requests = "*"
#
pytest = { version = "^4.6", optional = true }

[tool.poetry.extras]
test = ["pytest"]

# [tool.poetry.dev-dependencies]
# use 'test' extra instead

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

tox.ini

[tox]
envlist = test, scripts
isolated_build = true

[testenv]
basepython = python3.7
whitelist_externals =
    pytest
    bash
extras =
    test
commands =
    test: pytest
    scripts: bash -c 'for f in scripts/*.py; do python "$f"; done'

Answer 2

Assuming you have installed poetry and tox and pytest are dependencies in your pyproject.yml (notice poetry run, see https://python-poetry.org/docs/cli/#run):

[tox]
envlist = py37
isolated_build = True
skipsdist = True

[testenv]
whitelist_externals = poetry
commands=
    poetry run pytest

optionally you can make the install happen on running the tests by changing the last bit to (but then you will need to have tox installed outside of poetry, which could cause you issues down the line)

commands=
    poetry install
    poetry run pytest

Also depending on your root folder and where the tests are you can configure the path for tox to change directory to by adding

changedir = tests

In which case the whole file would look like this if you are in directory foo executing tox:

[tox]
envlist = py37
isolated_build = True
skipsdist = True

[testenv]
whitelist_externals = poetry
commands=
    poetry run pytest
changedir = tests

Answer 3

This is an easy fix. First, run rm -rf .tox to delete the .tox directory, probably hasn't installed the files you wanted.

Here is an example of my tox.ini.

[tox]
isolated_build = true
skipsdist = true
envlist = py39

[testenv]
deps = -rrequirements-dev.txt
whitelist_externals =
    poetry
skip_install = true
commands =
    python -m pytest tests/

As you can see in the tox.ini file I have a separate requirements for dev. This can be generated from poetry via the commands below.

poetry export --format=requirements.txt --without-hashes --with dev --output=requirements.txt

poetry export --format=requirements.txt --without-hashes --output=requirements-dev.txt

Don't ask me why but the deps in [testenv] are concatenated out, as you can see with deps = -rrequirements-dev.txt, would also change your command to python -m pytest /tests

Ciao.

Answer 4

First I need to install the dependencies with poetry install. Then append poetry run to the beginning of the command to enable the dependencies. Also running python scripts like that will just run the first, passing the name of the others as args to the first program. Instead use for f in scripts/*.py; do python "$f"; done (see here)

All together

poetry install
poetry run bash -c 'for f in scripts/*.py; do python "$f"; done'