Pytest - how to skip tests unless you declare an option/flag?

Question

I have some unit tests, but I'm looking for a way to tag some specific unit tests to have them skipped unless you declare an option when you call the tests.

Example: If I call pytest test_reports.py, I'd want a couple specific unit tests to not be run.

But if I call pytest -<something> test_reports, then I want all my tests to be run.

I looked into the @pytest.mark.skipif(condition) tag but couldn't quite figure it out, so not sure if I'm on the right track or not. Any guidance here would be great!

Answer 1

The pytest documentation offers a nice example on how to skip tests marked "slow" by default and only run them with a --runslow option:

# conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--runslow", action="store_true", default=False, help="run slow tests"
    )


def pytest_configure(config):
    config.addinivalue_line("markers", "slow: mark test as slow to run")


def pytest_collection_modifyitems(config, items):
    if config.getoption("--runslow"):
        # --runslow given in cli: do not skip slow tests
        return
    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

We can now mark our tests in the following way:

# test_module.py
from time import sleep

import pytest


def test_func_fast():
    sleep(0.1)


@pytest.mark.slow
def test_func_slow():
    sleep(10)

The test test_func_fast is always executed (calling e.g. pytest). The "slow" function test_func_slow, however, will only be executed when calling pytest --runslow.

Answer 2

We are using markers with addoption in conftest.py

testcase:

@pytest.mark.no_cmd
def test_skip_if_no_command_line():
    assert True

conftest.py:

def pytest_addoption(parser):
    parser.addoption("--no_cmd", action="store_true",
                     help="run the tests only in case of that command line (marked with marker @no_cmd)")

def pytest_runtest_setup(item):
    if 'no_cmd' in item.keywords and not item.config.getoption("--no_cmd"):
        pytest.skip("need --no_cmd option to run this test")

pytest call:

py.test test_the_marker  # test will be skipped
    
py.test test_the_marker --no_cmd  # test will run

Answer 3

There are two ways to do that:

First method is to tag the functions with @pytest.mark decorator and run / skip the tagged functions alone using -m option.

@pytest.mark.anytag
def test_calc_add():
    assert True

@pytest.mark.anytag
def test_calc_multiply():
    assert True

def test_calc_divide():
    assert True

Running the script as py.test -m anytag test_script.py will run only the first two functions.

Alternatively run as py.test -m "not anytag" test_script.py will run only the third function and skip the first two functions.

Here 'anytag' is the name of the tag. It can be anything.!

Second way is to run the functions with a common substring in their name using -k option.

def test_calc_add():
    assert True

def test_calc_multiply():
    assert True

def test_divide():
    assert True

Running the script as py.test -k calc test_script.py will run the functions and skip the last one.

Note that 'calc' is the common substring present in both the function name and any other function having 'calc' in its name like 'calculate' will also be run.

Answer 4

Following the approach suggested in the pytest docs, thus the answer of @Manu_CJ, is certainly the way to go here.

I'd simply like to show how this can be adapted to easily define multiple options:

The canonical example given by the pytest docs highlights well how to add a single marker through command line options. However, adapting it to add multiple markers might not be straight forward, as the three hooks pytest_addoption, pytest_configure and pytest_collection_modifyitems all need to be evoked to allow adding a single marker through command line option.

This is one way you can adapt the canonical example, if you have several markers, like 'flag1', 'flag2', etc., that you want to be able to add via command line option:

# content of conftest.py
 
import pytest

# Create a dict of markers.
# The key is used as option, so --{key} will run all tests marked with key.
# The value must be a dict that specifies:
# 1. 'help': the command line help text
# 2. 'marker-descr': a description of the marker
# 3. 'skip-reason': displayed reason whenever a test with this marker is skipped.
optional_markers = {
    "flag1": {"help": "<Command line help text for flag1...>",
             "marker-descr": "<Description of the marker...>",
             "skip-reason": "Test only runs with the --{} option."},
    "flag2": {"help": "<Command line help text for flag2...>",
             "marker-descr": "<Description of the marker...>",
             "skip-reason": "Test only runs with the --{} option."},
    # add further markers here
}


def pytest_addoption(parser):
    for marker, info in optional_markers.items():
        parser.addoption("--{}".format(marker), action="store_true",
                         default=False, help=info['help'])


def pytest_configure(config):
    for marker, info in optional_markers.items():
        config.addinivalue_line("markers",
                                "{}: {}".format(marker, info['marker-descr']))


def pytest_collection_modifyitems(config, items):
    for marker, info in optional_markers.items():
        if not config.getoption("--{}".format(marker)):
            skip_test = pytest.mark.skip(
                reason=info['skip-reason'].format(marker)
            )
            for item in items:
                if marker in item.keywords:
                    item.add_marker(skip_test)

Now you can use the markers defined in optional_markers in your test modules:

# content of test_module.py

import pytest


@pytest.mark.flag1
def test_some_func():
    pass


@pytest.mark.flag2
def test_other_func():
    pass

Answer 5

If the use-case prohibits modifying either conftest.py and/or pytest.ini, here's how to use environment variables to directly take advantage of the skipif marker.

test_reports.py contents:

import os
import pytest

@pytest.mark.skipif(
    not os.environ.get("MY_SPECIAL_FLAG"),
    reason="MY_SPECIAL_FLAG not set in environment"
)
def test_skip_if_no_cli_tag():
    assert True

def test_always_run():
    assert True

In Windows:

> pytest -v test_reports.py --no-header
================== test session starts ===================
collected 2 items

test_reports.py::test_skip_if_no_cli_tag SKIPPED    [ 50%]
test_reports.py::test_always_run PASSED             [100%]

============== 1 passed, 1 skipped in 0.01s ==============

> cmd /c "set MY_SPECIAL_FLAG=1&pytest -v test_reports.py --no-header"
================== test session starts ===================
collected 2 items

test_reports.py::test_skip_if_no_cli_tag PASSED     [ 50%]
test_reports.py::test_always_run PASSED             [100%]

=================== 2 passed in 0.01s ====================

In Linux or other *NIX-like systems:

$ pytest -v test_reports.py --no-header
================= test session starts =================
collected 2 items

test_reports.py::test_skip_if_no_cli_tag SKIPPED [ 50%]
test_reports.py::test_always_run PASSED         [100%]

============ 1 passed, 1 skipped in 0.00s =============

$ MY_SPECIAL_FLAG=1 pytest -v test_reports.py --no-header
================= test session starts =================
collected 2 items

test_reports.py::test_skip_if_no_cli_tag PASSED [ 50%]
test_reports.py::test_always_run PASSED         [100%]

================== 2 passed in 0.00s ==================

MY_SPECIAL_FLAG can be whatever you wish based on your specific use-case and of course --no-header is just being used for this example.

Enjoy.