How to improve branch coverage in C++

Question

I have a fairly large test suite for a C++ library with close to 100% line coverage, but only 55.3% branch coverage. Skimming through the results of lcov, it seems as if most of the missed branches can be explained by C++'s many ways to throw std::bad_alloc, e.g. whenever an std::string is constructed.

I was asking myself how to improve branch coverage in this situation and thought that it would be nice to have a new operator that can be configured to throw std::bad_alloc after just as many allocations needed to hit each branch missed in my test suite.

I (naively) tried defining a global void* operator new (std::size_t) function which counts down a global int allowed_allocs and throws std::bad_alloc whenever 0 is reached.

This has several problems though:

It is hard to get the number of new calls until the "first" desired throw. I may execute a dry run to calculate the required calls to succeed, but this does not help if several calls can fail in the same line, e.g. something like std::to_string(some_int) + std::to_string(another_int) where each std::to_string, the concatenation via operator+ and also the initial allocation may fail.
Even worse, my test suite (I am using Catch) uses a lot of new calls itself, so even if I knew how many calls my code needs, it is hard to guess how many additional calls of the test suite are necessary. (To make things worse, Catch has several "verbose" modes which create lots of outputs which again need memory...)

Do you have any idea how to improve the branch coverage?

Update 2017-10-07

Meanwhile, I found https://stackoverflow.com/a/43726240/266378 with a link to a Python script to filter some of the branches created by exceptions from the lcov output. This brought my branch coverage to 71.5%, but the remaining unhit branches are still very strange. For instance, I have several if statements like this:

with four (?) branches of which one remained unhit (reference_token is a std::string).

Does anyone has an idea what these branches mean and how they can be hit?

@NeilButterworth Catch has an assertion `CHECK_THROWS_AS` which succeeds if the passed expression throws a specific exception. — Niels Lohmann, Oct 04 '17 at 22:27
@MooingDuck Yes, most missed branches are in lines without if/switch, but simple assignments to functions that may throw because they call constructors. — Niels Lohmann, Oct 04 '17 at 22:30
@Neils I don't see that as recovery, only reporting, which would probably be done for you by your runtime or OS even if you didn't handle it. My point is, if you can't recover, it probably isn't worth testing - just fail. — , Oct 04 '17 at 22:35
@NeilButterworth: I keep hearing that, but if your code has any reasonable exception guarantees, then stack unwinding will have destroyed a lot, so you can simply catch the `bad_alloc` and show an error to the user. I've worked on several apps that had no issues doing that. — Mooing Duck, Oct 04 '17 at 22:39
@Mooing Once you have a bad_alloc you are in a very grey area where producing an error message is moot (because producing the error message may also cause a bad_alloc).. I would prefer a semi-robust fail and exit. Also, in my experience a bad_alloc in real code is incredibly rare - I don't think I've ever seen one. — , Oct 04 '17 at 22:44
@NeilButterworth: Specifically, I worked on one project that dealt with large audio files (hours long) and an Android app that occasionally dealt with large image files. In both cases, unwinding the stack for "the current task" feed enough memory that we have never had a subsequent issue showing the message. — Mooing Duck, Oct 04 '17 at 22:46
It sounds like lcov is noticing the exception propagation logic required to support the allocator, because it *can* throw that exception. Why don't you try a version of new that doesn't thow the exception? YOu might have to catch the exception inside *inside* your funny version of new and supress it. (Awful hack: when you catch the exception, return the address of a static buffer). — Ira Baxter, Oct 10 '17 at 13:49

score 1 · Answer 1 · answered Oct 05 '17 at 06:06

1

Whose code are you wanting to test - yours or the Standard Library? It strikes me that your coverage report is telling you about branches in 'std::string' rather than your code.

Can you configure 'lcov' to ignore the std library and just focus on your code?

answered Oct 05 '17 at 06:06

JBRWilkinson

4,821
1
24
36

1

I did not find a way to achieve this. – Niels Lohmann Oct 07 '17 at 17:32
2

`--exclude-throw-branches` flag can help for `gcovr` – Oleh Pomazan May 22 '20 at 14:49
1

@OlehPomazan I faced the same issue (had 54% coverage due to branch coverage being low). Your solution fixed my issue and improved to 87%. Thank you so much! My command to make it work: `gcovr -r . --exclude-throw-branches --xml -o coverage.xml` – lbragile Jun 13 '20 at 20:20

score 0 · Answer 2 · answered Oct 04 '17 at 22:45

I had a successful go at this a while back. I didn't have a test suite, just ran my application, but found the following.

Some form of isolation of the thing under test was important. I had vectors and maps, which basically interrupted test when they were also prone.

I think I succeeded when I had an IPC between the fault injection and the failure point. That allowed the fault injector code to new and delete independently of the thing under test

I have also succeeded with similar things when in the same binary, but having two independent allocation paths - a custom allocator for the fault injection code - which ensures it does not get interfered with.

My successful system took a call-stack of a malloc and sent it through IPC to another program. It decided if it had seen the stack before, and if not, it failed the allocation. Then the program might crash and fail (the core-dump was captured), then the test system re-launched. This dramatically improved the quality of the code I was developing.

How to improve branch coverage in C++

Update 2017-10-07

2 Answers2