Using py.test with coverage doesn't include imports

Question

For Jedi we want to generate our test coverage. There is a related question in stackoverflow, but it didn't help.

We're using py.test as a test runner. However, we are unable to add the imports and other "imported" stuff to the report. For example __init__.py is always reported as being uncovered:

Name                           Stmts   Miss  Cover
--------------------------------------------------
jedi/__init__                      5      5     0%
[..]

Clearly this file is being imported and should therefore be reported as tested.

We start tests like this [*]:

py.test --cov jedi

As you can see we're using pytest-coverage.

So how is it possible to properly count coverage of files like __init__.py?

[*] We also tried starting test without --doctest-modules (removed from pytest.ini) and activate the coverage module earlier by py.test -p pytest_cov --cov jedi. Neither of them work.

I've offered a bounty. Please try to fix it within Jedi. It's publicly available.

do you have a conftest.py file that imports "jedi" by chance? I am guessing that after pytest initialization (which includes reading of conftest.py files at CWD and test*/ if it exists) your jedi has already been imported. — hpk42, May 07 '13 at 15:53
Yes. We have a `conftest.py`. It indirectly imports `jedi`. However, moving these indirect imports into the functions, also didn't help (or is that also a problem???). The file is here: https://github.com/davidhalter/jedi/blob/master/test/conftest.py — Dave Halter, May 07 '13 at 17:44
What is *in* `__init__.py`? Importing a function us not the same as running it, for example. — Martijn Pieters, May 11 '13 at 09:33
This is our `__init__.py` https://github.com/davidhalter/jedi/blob/master/jedi/__init__.py — Dave Halter, May 12 '13 at 09:51

Dave Halter · Accepted Answer · 2014-04-29T13:56:24.607

79

@hynekcer gave me the right idea. But basically the easiest solution lies somewhere else:

Get rid of pytest-cov!

Use

coverage run --source jedi -m py.test
coverage report

instead!!! This way you're just running a coverage on your current py.test configuration, which works perfectly fine! It's also philosophically the right way to go: Make each program do one thing well - py.test runs tests and coverage checks the code coverage.

Now this might sound like a rant, but really. pytest-cov hasn't been working properly for a while now. Some tests were failing, just because we used it.

As of 2014, pytest-cov seems to have changed hands. py.test --cov jedi test seems to be a useful command again (look at the comments). However, you don't need to use it. But in combination with xdist it can speed up your coverage reports.

edited Apr 29 '14 at 13:56

answered May 13 '13 at 14:23

Dave Halter

15,556
13
76
103

2

There is a development version of pytest-cov which correctly collects coverage for conftest: https://github.com/schlamar/pytest-cov/pull/4. It looks good running against jedi: https://gist.github.com/schlamar/7722c21c3f2f1b97c920. Please feel free to submit further issues (I just started maintaining this package). – schlamar Apr 24 '14 at 11:03
3

@schlamar: The question is: Why would we even need a `pytest-cov` package if the commands above work perfectly well? – Dave Halter Apr 25 '14 at 11:35
3

support for xdist which reduces test duration by almost 50%: https://gist.github.com/schlamar/7722c21c3f2f1b97c920#file-gistfile2-txt + less commands to type and remember – schlamar Apr 25 '14 at 22:42

score 9 · Answer 2 · edited Feb 04 '20 at 14:24

I fixed the test coverage to 94% by this patch that simplifies import dependencies and by the command:

py.test --cov jedi test                    # or
py.test --cov jedi test --cov-report=html  # + a listing with red uncovered lines

Uncovered lines are only in conditional commands or in some less used functions but all headers are completely covered.

The problem was that the tests configuration test/conftest.py did import prematurely by dependencies almost all files in the project. The conftest file defines also additional command line options and settings that should be set before running the test. Therefore I think that pytest_cov plugin works correctly if it ignores everything that was imported together with this file, although it is a pain. I excluded also __init__.py and settings.py from the report because they are simple and with the complete coverage but they are also imported prematurely in the dependency of conftest.

As discussed on github (https://github.com/davidhalter/jedi/pull/214) you'll get the bounty, because you gave me the right idea. however, this didn't work. — Dave Halter, May 13 '13 at 14:24

Jan Vlcinsky · Answer 3 · 2016-01-22T14:10:46.183

In my case, all the tests run, but coverage was 0%.

The fix was:

$ export PYTHONPATH="."

After the results were correct.

I had in past few problems with py.test command having problems to import something and setting the PYTHONPATH env var was the solution. It worked for me this time too.

My real example with awslogs

First with PYTHONPATH unset:

$ py.test --cov=awslogs  tests/
========================================= test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/javl/sandbox/awslogs/github/awslogs, inifile:
plugins: cov-2.2.0
collected 11 items

tests/test_it.py ...........Coverage.py warning: No data was collected.

--------------------------- coverage: platform linux2, python 2.7.9-final-0 ---------------------------
Name                    Stmts   Miss  Cover
-------------------------------------------
awslogs/__init__.py         2      2     0%
awslogs/bin.py             85     85     0%
awslogs/core.py           143    143     0%
awslogs/exceptions.py      12     12     0%
-------------------------------------------
TOTAL                     242    242     0%

====================================== 11 passed in 0.38 seconds ======================================

Resulting coverage is 0%.

Then I set the PYTHONPATH:

$ export PYTHONPATH="."

and rerun the test:

$ py.test --cov=awslogs  tests/
========================================= test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/javl/sandbox/awslogs/github/awslogs, inifile:
plugins: cov-2.2.0
collected 11 items

tests/test_it.py ...........
--------------------------- coverage: platform linux2, python 2.7.9-final-0 ---------------------------
Name                    Stmts   Miss  Cover
-------------------------------------------
awslogs/__init__.py         2      0   100%
awslogs/bin.py             85      9    89%
awslogs/core.py           143     12    92%
awslogs/exceptions.py      12      2    83%
-------------------------------------------
TOTAL                     242     23    90%

====================================== 11 passed in 0.44 seconds ======================================

Now is the coverage 90%.

WARNING: Manipulating PYTHONPATH can have strange side effects. Currently I run into problem, that pbr based package is creating egg directory when building distributable and if PYTHONPATH is set to ".", it automatically considers the egg related package as installed. For this reason I stopped using pytest-cov and follow the advice to use coverage tool instead.

Do you know `pip install -e .`? This allows you to install a development version of your software and might be the issue that you have here (not being included in the `sys.path`). — Dave Halter, Feb 02 '16 at 15:39
@DaveHalter Thanks, good idea. Sure I know `pip install -e .` However when installing by `pip install .` (without the `-e` option), the plain `coverage` command handles the coverage properly but `py.test --cov==awslogs -m py.test -sv tests` does not. — Jan Vlcinsky, Feb 02 '16 at 18:19
@JanVlcinsky (or those reading this thread). Coverage doesn't handle the default python behaviour of adding the current working directory to sys.path, therefore `coverage run --source mypackage -m pytest tests` will import the local `mypackage` directory instead of that installed by `pip install .` (unless you're using a `src` folder, see https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure for why this is a good idea). `pytest-cov` explicitly avoids this by removing the cwd from `sys.path` so this doesn't happen. — Will, Jan 01 '19 at 22:22
My folder structure is `src/ src/tests`. Setting `PYTHONPATH` to `$(pwd)/src` fixed the test coverage issue for me. — Marcello Romani, Jun 12 '19 at 13:39

score 3 · Answer 4 · answered Aug 07 '13 at 13:21

I had this problem with py.test, the coverage and the django plugin. Apparently the model files are imported before coverage is started. Not even "-p coverage" for early-loading of the coverage-plugin worked.

I fixed it (ugly?) by removing the models module from sys.modules and re-importing it in the test file that tests the model:

import sys
del sys.modules['project.my_app.models']
from project.my_app import models

def test_my_model():
  ...

score 1 · Answer 5 · answered Apr 29 '20 at 07:33

1

if you are using flask then this will help you to resolve the issue-

pytest --cov=src --cov-report=html

answered Apr 29 '20 at 07:33

Sachin

166
1
8

Using py.test with coverage doesn't include imports

5 Answers5

Linked