Problems with code coverage

One important testing technique is to monitor how the application is covered by tests. We track how many use cases we have covered, how many customer requests, but the most accurate at least in terms of functionality is code coverage. In determining the code coverage it is recorded what code was run during tests and what has not yet been tested.

Simplest but inadequate and misleading is statement coverage – command coverage.

It only checks whether the command line was executed during testing, to cover conditions it is enough to execute it with any of its evaluation.

Formally: The test set T satisfies the criterion of coverage of commands for a given code K if for every command p belonging to the code K there is a test t from the set T, that during execution of test t will be executed command p.

For tester or programmer is not a problem to achieve very high (even 100%) command coverage of a very buggy program without a single failed test, without a single mistake being discovered.
In spite of this metric is misleading, it is unfortunately popular for its simplicity.

More advanced metric is decision coverage (also branch coverage).

Formally: Consider a graph, where the statements are nodes and transitions between the edges. Then consecutive statements are edge and condition is node of which lead two edges, one for true and one for false value. Then the test set T satisfies the decision coverage criterion for the code K if for each edge h of the mentioned graph, there is a test t from test set T that during execution of test t the run goes through edge h.

Simply put, with decision coverage each condition is node with two edges leading from it.

For example:

We have code with three conditions, one of which is nested.



Then decision coverage graph looks like this:



Blue circles are conditions, green ones are statements.

Even here we can achieve full coverage without the discovery of many errors.

The next stage is condition coverage.

Formally: A set of tests T satisfies the condition coverage criterion for the code K if it satisfies the decision coverage criterion and for each part p of each composite condition, there are tests t and u from the set T that during execution of t p is evaluated as true and during execution of u as false.

If you skip the part where condition coverage has to satisfy the decision coverage criterion, then the case of being not covered by decisions but being covered by conditions occurs when for all possible inputs a condition is evaluated always as positive or negative.

Improvement from decision coverage is that condition coverage takes into account all possible evaluations of conditions, not only if it is true or not.

For example:
Let's have complex condition if (a>0 || b>0).
It takes these two tests to reach decision coverage: {a = 5; b = -2}, {a = 0; b = 0}
These tests do not reach condition coverage because b > 0 is false in both tests.
We need two different tests to reach condition coverage: {a = 5; b = 2}, {a = 0; b = 0}

In this version it is more difficult to find a bug that would not be revealed if code has full condition coverage, although certain types of bugs will still remain undetected even in fully covered code.

The highest level of coverage is path coverage. It finds out whether each of the possible paths in each function has been run and therefore it means a very detailed testing.

Formally: A set of tests T satisfies the path coverage criterion for the code K, if it satisfies condition coverage criterion and for each path C linking the input and output node in the graph of code and containing at most n cycles, there is a test t from the test set T that during execution of t the run goes through path C.

While you cannot detect all bugs only from the code, path coverage provides assurance that all options of the run have been tested.
The problem is the practical inapplicability of this coverage, since its complexity causes an exponential increase in the number of tests.

Try these exercises:

Question 1:
How many tests (runs of code) we need to have a method code_coverage covered by statements; decisions; conditions; paths?



Question 2:
How many tests (runs of code) we need to have a method code_coverage2 covered by statements; decisions; conditions; paths?