How can I combine coverage results with tox?

Question

For my mpu package I have execution-environment dependend code like

if sys.version_info < (3, 0):
   pass  # do something
else:
   pass  # do something else

and a tox file

[tox]
envlist = py27,py36

[testenv]
deps =
    pytest
    pytest-cov
    pytest-pep8
    pydocstyle
commands =
    pip install -r test-requirements.txt
    pip install -e .[all]
    pytest .
    pydocstyle

and a setup.cfg

[metadata]
description-file = README.md

[tool:pytest]
addopts = ./tests/ --doctest-modules --cov=./mpu --cov-report html:tests/reports/coverage-html --cov-report xml:tests/reports/coverage.xml --pep8 --ignore=docs/
doctest_encoding = utf-8

[pydocstyle]
ignore = D104, D105, D107, D301, D413, D203, D212, D100
match_dir = mpu

The tox file seems to do what I want, but the coverage is only for one of the tested environments. I have seen Reporting cumulative coverage across multiple Python versions in the branch coverage-combinedcoverage-combined, but it doesn't work. For the first run, it seems as if it didn't execute all tests as the test coverage was way lower than before. My guess is that doctests were not executed. For the second run, I get

ERROR: InvocationError: '/home/moose/GitHub/mpu/.tox/py27/bin/coverage run --source=mpu/ setup.py test'

System

$ coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io

Errors

Now I get this error:

======================================================================
ERROR: test_pd (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_pd
Traceback (most recent call last):
  File "/usr/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/moose/GitHub/mpu/tests/test_pd.py", line 8, in <module>
    from mpu.pd import example_df
  File "/home/moose/GitHub/mpu/mpu/pd.py", line 10, in <module>
    import pandas as pd
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module>
    from pandas.core.api import *
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
    from pandas.core.groupby.groupby import Grouper
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 2, in <module>
    from pandas.core.groupby.groupby import (
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 49, in <module>
    from pandas.core.frame import DataFrame
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 74, in <module>
    from pandas.core.series import Series
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/core/series.py", line 80, in <module>
    import pandas.plotting._core as gfx
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/plotting/__init__.py", line 11, in <module>
    from pandas.plotting._core import boxplot
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/plotting/_core.py", line 45, in <module>
    from pandas.plotting import _converter
  File "/home/moose/.local/lib/python3.6/site-packages/pandas/plotting/_converter.py", line 8, in <module>
    import matplotlib.units as units
  File "/home/moose/.local/lib/python3.6/site-packages/matplotlib/__init__.py", line 131, in <module>
    from matplotlib.rcsetup import defaultParams, validate_backend, cycler
  File "/home/moose/.local/lib/python3.6/site-packages/matplotlib/rcsetup.py", line 29, in <module>
    from matplotlib.fontconfig_pattern import parse_fontconfig_pattern
  File "/home/moose/.local/lib/python3.6/site-packages/matplotlib/fontconfig_pattern.py", line 22, in <module>
    from pyparsing import (Literal, ZeroOrMore, Optional, Regex, StringEnd,
  File "/home/moose/.local/lib/python3.6/site-packages/pyparsing.py", line 943, in <module>
    collections.MutableMapping.register(ParseResults)
  File "/usr/lib/python3.6/abc.py", line 158, in register
    if issubclass(subclass, cls):
  File "/usr/lib/python3.6/abc.py", line 228, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.6/abc.py", line 228, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.6/typing.py", line 1154, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/usr/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/usr/lib/python3.6/typing.py", line 884, in __extrahook__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.6/typing.py", line 1154, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/usr/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/usr/lib/python3.6/typing.py", line 884, in __extrahook__
[...]
  File "/usr/lib/python3.6/typing.py", line 884, in __extrahook__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.6/abc.py", line 206, in __subclasscheck__
    elif subclass in cls._abc_negative_cache:
  File "/usr/lib/python3.6/_weakrefset.py", line 75, in __contains__
    return wr in self.data
RecursionError: maximum recursion depth exceeded in comparison

Is this resolved? Not clear from the section in the end. Can you just clarify? — Tarun Lalwani, Jun 22 '18 at 07:04
No, it is not resolved. I just found it confusing that `--parallel-mode` has to be before the setup.py and wanted to note it. Maybe I should remove this part from the question. — Martin Thoma, Jun 22 '18 at 07:16
See if this repo is of any help? https://github.com/avara1986/gozokia — Tarun Lalwani, Jun 22 '18 at 07:33
Ah, ok, that is another way. But I hoped I could do it with tox. I would prefer to have `tox` as the command to execute everythin related to tests — Martin Thoma, Jun 22 '18 at 07:40
I think the pattern with `begin,py{...},end` is suitable for this, but I don't understand why/how this causes the `ERROR: InvocationError` — Martin Thoma, Jun 22 '18 at 07:42

score 9 · Accepted Answer · edited Jun 05 '21 at 08:36

9

I'm assuming that you have pytest-cov installed. In that case, it's fairly simple to append the coverage. You'll simply need to add --cov-append to the [tool:pytest].addopts section.

[tool:pytest]
addopts = tests/ --doctest-modules --cov=./mpu --cov-append --cov-report html:tests/reports/coverage-html --cov-report xml:tests/reports/coverage.xml --pep8 --ignore=docs/
doctest_encoding = utf-8

This will combine the coverage from multiple test runs together. This is also helpful if you split out integration/unit tests.

edited Jun 05 '21 at 08:36

webknjaz -- Слава Україні

687
4
34

answered Jun 25 '18 at 01:08

Tim Martin

2,419
1
21
25

While I appreciate your answer (+1), it seems not to solve the problem(s). See bug I added to the question. – Martin Thoma Jun 25 '18 at 20:40
@MartinThoma the error you posted is unrelated to the `--cov-append` option - for some reason, tests are run with `unittest` instead of `pytest`. you must have changed something else to cause that. – hoefling Jun 26 '18 at 11:53
1

Please add how you figured out that `unittest` was the problem. I could fix it now: I called `coverage` directly instead of using `pytest` and let `pytest-cov` call coverage. Thank you! – Martin Thoma Jun 26 '18 at 18:07

How can I combine coverage results with tox?

System

Errors

1 Answers1