30

I would like to have a list of all the tests that have failed to be used at the end of session.

Pytest lets you define a hook pytest_sessionfinish(session, exitstatus), that is called at the end of the session, where I wish to have that list.

session is a _pytest.main.Session instance that has the attribute items (type list), but I couldn't find whether the each item in that list passed of failed.

  1. How can a list of all failed tests could be retrieved at the end of the session?

  2. How can it be done while using pytest-xdist plugin, where I would like to get that list in the master process. Using this plugin, session does not even have items attribute in the master:

    def pytest_sessionfinish(session, exitstatus):
    if os.environ.get("PYTEST_XDIST_WORKER", "master") == "master":
         print(hasattr(session, "items"))  # False
Itay
  • 335
  • 1
  • 3
  • 6

7 Answers7

40

Run pytest with -rf to get it to print a list of failed tests at the end.

From py.test --help:

  -r chars              show extra test summary info as specified by chars
                        (f)ailed, (E)error, (s)skipped, (x)failed, (X)passed,
                        (p)passed, (P)passed with output, (a)all except pP.
                        Warnings are displayed at all times except when
                        --disable-warnings is set

Here's what you get:

$ py.test -rf
================= test session starts =================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.6.0, pluggy-0.7.1
[...]
=============== short test summary info ===============
FAILED test_foo.py::test_foo_is_flar
FAILED test_spam.py::test_spam_is_mostly_pork
FAILED test_eggs.py::test_eggs_are_also_spam
=== 3 failed, 222 passed, 8 warnings in 12.52 seconds ==
Aaron V
  • 6,596
  • 5
  • 28
  • 31
  • I use `pytest ... -rA` and it gives failures output before passed output, is there a way to make it give failures output at the end? So less scrolling up – cryanbhu Nov 25 '20 at 06:34
5

--result-log is deprecated. You can instead use -v to output the test case names as they run. If you pipe that into a file, you can query it. So if you're running your tests from a script you can do something like:

pytest -v | tee log.txt
grep -E '::.*(FAILURE|ERROR)' log.txt
z0r
  • 8,185
  • 4
  • 64
  • 83
4

If you want results of the tests you can use hook runtest_makereport:

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()
    if rep.when == 'call' and rep.failed:
        mode = 'a' if os.path.exists('failures') else 'w'
        try:  # Just to not crash py.test reporting
          pass  # the test 'item' failed
        except Exception as e:
            pass
ANDgineer
  • 505
  • 3
  • 12
  • 3
    Thanks! That seems to work well when xdist is not in use, and I can save the failed tests to a global variable to be used by `pytest_sessionfinish` at the end. **When using xdist**, `pytest_runtest_makereport` is called only for the workers and not for master, so they can communicate through file, but a better solution is using `pytest_report_teststatus` and `pytest_runtest_logreport` that **are called in the master** and can be used to collect failed test. – Itay Jan 03 '18 at 21:51
3

You can use command line option --result-log:

test_dummy.py:

def test_dummy_success():
    return


def test_dummy_fail():
    raise Exception('Dummy fail')

Command line:

$ py.test --result-log=test_result.txt

Content of the test_result.txt

. test_dummy.py::test_dummy_success
F test_dummy.py::test_dummy_fail
 def test_dummy_fail():
 >       raise Exception('Dummy fail')
 E       Exception: Dummy fail

 test_dummy.py:6: Exception

Just search 'F' in first column and after that would be [file]::[test]

ANDgineer
  • 505
  • 3
  • 12
  • 1
    I would like to use this information in `pytest_sessionfinish` and this method means failed tests list is available only after the pytest process has exited. – Itay Jan 03 '18 at 21:53
3

I wanted a concise report of failed tests and parametrized variations so went with pytest_terminal_summary in conftest.py:

def pytest_terminal_summary(terminalreporter, exitstatus, config):
    terminalreporter.section('Failed tests')
    failures = [report.nodeid.split('::')[-1]
                for report in terminalreporter.stats.get('failed', [])]
    terminalreporter.write('\n'.join(failures) + '\n')

If you inspect terminalreporter._session.items, there's more information you can add to the report, this is just what I wanted.

Joao Coelho
  • 2,838
  • 4
  • 30
  • 36
1

You can get the details of only failing tests and save the logs into a file with command below. Logs also contain the trace of each test.

py.test -rf tests/ | tee logs.txt
abdullahselek
  • 7,893
  • 3
  • 50
  • 40
-1

I came access this question, trying to figure out internal data structure for the session instance.

Iterating over session.items you can check rep_call.outcome for string representation of the test result.

def pytest_sessionfinish(session, exitstatus):
    for item in session.items:
        print('{} {}'.format(item.name, item.rep_call.outcome))

With trivial test cases you get this

test_positive passed
test_negative failed
Aleks Amirkhanov
  • 1
  • 3