Assert no infinite looping in pytest

Question

I have a function that should not hang. Specifically, it detects the condition that would cause it to hang and raises an error instead. Simplified, my function is similar to this one:

def chew_repeated_leading_characters(string: str, leader: str):
    if len(leader) == 0:
        # Without this, will hang on empty string
        raise ValueError('leader cannot be empty')

    while True:
        if string.startswith(leader):
            string = string[len(leader):]
        else:
            break

    return string

I would like to write a unit test in pytest to ensure this behavior. I know how to assert that a test should raise an exception. However, in this case, if the test fails, it does not actually fail; the entire test apparatus simply hangs.

How do I generate a test failure on endless looping?

Please include some code showing what you're trying t test. Without it, the only real answer is "no". — Jonathon Reinhart, Apr 23 '18 at 10:28
See my edit. Done without any timeouts. You can add a timeout if you want to be 100% sure that no automation will hang, but IMO it would be an overkill. — Błażej Michalik, Apr 23 '18 at 10:45
You shouldn't use `assert` in 'real' code, you should raise `ValueError` instead. See https://stackoverflow.com/a/1838411/2372812 and http://archive.is/5GfiG — Tom Dalton, Apr 23 '18 at 10:49
@TomDalton he's talking about an unit test written in pytest, not production code. — Błażej Michalik, Apr 23 '18 at 10:50
I believe the function code shown above (`chew_repeated_leading_characters`) is the function he wants to test, it isn't the test function itself. — Tom Dalton, Apr 23 '18 at 10:52
@TomDalton Oh, sorry, missed the point. Yes, `ValueError` is more appropriate here. — Błażej Michalik, Apr 23 '18 at 10:57
@TomDalton This is just a toy example. I'll change it anyway so that people arriving here don't pick up bad habits. — drhagen, Apr 23 '18 at 10:59
Good stuff! You've certainly prompted an interesting discussion on different ways to handle this situation! — Tom Dalton, Apr 23 '18 at 11:01

Błażej Michalik · Answer 1 · 2018-04-26T18:15:35.927

Such assertion does not exist. A 'no infinite looping' assertion would be a solution to the Halting Problem, which is proven to have none.

You should instead aim to check whether an event after the raise clause happened, and fail if it did. If your function uses a side effect after that raise clause, it would be better to check for that side effect, instead of adding arbitrary timeouts to the tests.

For example, this method:

def method(self, should_loop=False):
    if should_loop:
        raise Exception("You really want me to loop forever?")
    while should_loop:
        self.side_effect()

Would be testable by mocking the side_effect(), and making it trigger a fail upon a call.

Timeouts should be used as a solution of last resort.

Edit: in respect to your code, it is easy to do it without a timeout - pass a mocked string, that triggers a failure upon a call to startswith():

from mock import Mock

...

string_mock = Mock()
string.startswith.side_effect = AssertionError('This will hang.')
leader_mock = Mock()
leader.__len__ = lambda self: 0
chew_repeated_leading_characters(string_mock, leader_mock)

The problem with mocking in this case is that if I ever change the internal implementation (say, to use a function other than `startswith`), the test will become defective without warning. — drhagen, Apr 23 '18 at 10:56
@drhagen it should be easy to make the exception trigger on any other call to the string or leader. I wouldn't stress about it that much though - after all, you can add `while True: pass` anywhere in your code, and no test will shield you from it. If you are worried about that happening, then yes, the only way to shield yourself from it is to use a timeout. The timeout should be a measure against the test hanging the automation though, not a correctness check. — Błażej Michalik, Apr 23 '18 at 11:01

score 1 · Answer 2 · answered Apr 23 '18 at 10:21

1

It's mathematically impossible to assert this. See the Halting Problem.

What you can do, however, is setup an alarm clock to go off after some maximum amount of time. If the test is still running, then it is a failure.

Without any code to look at, it's hard to make a more clear suggestion.

answered Apr 23 '18 at 10:21

Jonathon Reinhart

132,704
33
254
328

Adding timeouts to unit tests is not a good idea. It calls for a redesign of the code. – Błażej Michalik Apr 23 '18 at 10:24
I agree. But just like you said, we have no idea what the OP's code is doing, so it's impossible to make a better suggestion. – Jonathon Reinhart Apr 23 '18 at 10:27

score 1 · Answer 3 · answered Apr 23 '18 at 10:24

1

While the above answers are correct that it's impossible to do this in the general sense, you could use something like https://pypi.org/project/pytest-timeout/ to achieve a practical solution.

answered Apr 23 '18 at 10:24

Tom Dalton

6,122
24
35

1

I would advise against adding timeouts to your tests. It's rarely a good idea. – Błażej Michalik Apr 23 '18 at 10:25
Well you can remove them again once you solve the halting problem :) – Tom Dalton Apr 23 '18 at 10:27
Or after you'll get angry enough of your CI VM failing the tests randomly. – Błażej Michalik Apr 23 '18 at 10:28
This is pretty much what I was looking for. Write a paper about how you solved the halting problem by forcing the thread to halt after a given amount of time. ;-) – drhagen Apr 23 '18 at 10:30
The tests wont fail 'randomly' if you use a sensible value for the timeout with your *known test data*. – Tom Dalton Apr 23 '18 at 10:31
Sure, I agree. But there are good solutions to this problem. Timeouts aren't one. See my answer. – Błażej Michalik Apr 23 '18 at 10:33
1

I looked into the source code of pytest-timeout. It uses `ITIMER_REAL`, so it is not measuring CPU time, but wall time. If the CI VM pauses the testing occasionally, there could be sporadic errors. – drhagen Apr 23 '18 at 11:16
Yes, pytest-timeout was designed to help you with deadlocking code on a CI server or such. That is, use it with a timeout in the range of minutes, not seconds. Using it for short timeouts is inherently brittle. – flub Apr 23 '18 at 18:04

Assert no infinite looping in pytest

3 Answers3