#What is coverage
Coverage is a tool that, when used with unit tests, will tell you what lines of code have actually been exercised by the tests. This will allow you to write tests to specifically cover parts of your code that are not being exercised by the tests.
Alternatively parts of your code that are not exercised by the tests may, in fact, be redundant. This might be especially true in a large evolved code base where old code has just been left laying around and, as it is not causing any errors, has not been removed.
#Using coverage
I am presently developing a pure lua XML parser (called plxml.lua). Along with the unit tests I want to know what parts of my code might be missed by the tests. So I have a simple script run the tests and report the coverage:
#!/bin/sh
WHERE=/usr/local/share/lua/5.1/
# Generate coverage stats
lua $WHERE/coverage.lua plxml.lua tests/*
# Report coverage stats
lua $WHERE/report_coverage.lua plxml.lua > report.txt
rm coverage.out
tail -n 5 report.txt
echo
echo Full report in report.txt
The coverage statistics are gathered by:
lua $WHERE/coverage.lua plxml.lua tests/*
this runs all the tests in the tests directory and records all the statistics against the plxml.lua. These results are then written out to a file called 'coverage.out'.
To generate the report:
lua $WHERE/report_coverage.lua plxml.lua > report.txt
This reads the 'coverage.txt' file and the source file (plxml.lua) and creates a report showing the lines of code that were exercised by the tests and those that were not along with some summary statistics at the end.
#Running a simple example
In the examples directory there are some tests we can run to show the usage of the coverage. First lets generate coverage for simple.lua:
lua ../coverage.lua simple.lua
This executes simple.lua and collects statistics. To view the report:
lua ../report_coverage.lua simple.lua
1 true function odd( number )
2 false if( number % 2 == 1 ) then
3 false return true
4 true else
5 false return false
6 true end
7 true end
8
9 true function nextnumber( number )
10 true if( number % 2 == 1 ) then
11 true return ( number * 3 ) + 1
12 true else
13 true return number / 2
14 true end
15 true end
16
17 true function solve( number )
18 true print(number)
19 true while( number > 1 ) do
20 true number = nextnumber( number )
21 true print(number)
22 true end
23 true end
24
25 true solve( 50 )
Total lines in file ..: 25
Total lines of code ..: 22
Code covered .........: 19 (86.36%)
Code missed ..........: 3
The first column is the line number from the source file, the second column is either true, false or blank. Blank means that it was that lines was either blank in the source file or a comment. True means that this was a line of code in the source that was executed during the tests end false means that it was not.
Reading the report we can see that the function odd() was not actually executed. Although the 'function odd ( number )', the 'else' and 'end's are marked as true the real code, on lines 2, 3 and 5, has not been run. This is because it is not being called. it should have been used in the if conditional at line 10.
It might seem strange that the 'function ...', 'else' and 'end's were flagged as true but the code was not actually run. This is because lua reports the 'function ...' as being executed when it compiles the source. The 'else' and 'end's are marked as true by the coverage reporter as this it is what I feel is right, lua itself does not report them as executed.
So we have a choice, remove the odd() function or use it at line 10.
#Things to look out for
The source even.lua shows two implementations of a function to check if a number is even. Although both are functionally equivalent evenbad() is written in a way, on a single line, that makes coverage reporting hard. Lets run them:
lua ../coverage.lua even.lua even.test.lua
lua ../report_coverage.lua even.lua
1 true function evenbad( number )
2 true if( number % 2 == 0 ) then return "even" else return "odd" end
3 true end
4
5 true function evengood( number )
6 true if( number % 2 == 0 ) then
7 true return "even"
8 true else
9 false return "odd"
10 true end
11 true end
Total lines in file ..: 11
Total lines of code ..: 10
Code covered .........: 9 (90.00%)
Code missed ..........: 1
From even.test.lua it is clear that we are only testing even numbers. But the coverage report of evenbad(), lines 1 to 3, would seem to say that the function has full coverage. This is not so if you consider the coverage of evengood(), lines 5 to 11.
This is a limitation of this coverage tool which is based on a debug hook that reports each time a line is executed. This it is unlikely to change unless something significant happens to Lua to allow greater granularity. You will have to look out for it.
The source conditional.lua show something else to watch out for. First we run the report:
lua ../coverage.lua conditional.lua
lua ../report_coverage.lua conditional.lua
1 true function odd( number )
2 true if( number % 2 == 1 ) then
3 true return true
4 true else
5 true return false
6 true end
7 true end
8
9 true function even( number )
10 false return not odd( number )
11 true end
12
13 true function special( number )
14 false return number == 42
15 true end
16
17 true function other( a, b )
18 true if( odd( a ) and even( b ) ) then
19 false return "yes"
20 true else
21 true return "no"
22 true end
23 true end
24
25 true function another( a, b )
26 true if( odd( a ) or special( b ) ) then
27 true return "yes"
28 true else
29 false return "no"
30 true end
31 true end
32
33 true other( 2, 3 )
34 true other( 2, 2 )
35 true other( 2, 1 )
36
37 true another( 1, 1 )
38 true another( 1, 42 )
39 true another( 1, 2 )
Total lines in file ..: 39
Total lines of code ..: 33
Code covered .........: 29 (87.88%)
Code missed ..........: 4
From lines 9 to 11 it is clear that the function even() was not called but the only line that calls it, the conditional on line 18, is marked as exercised. This is because of the 'and' operator. The left hand side, the odd(), always fails as it is always called with an even number. Lua knows the conditional has failed and even() is never called.
Likewise the function special() on lines 13 to 15 is never called. This is because the 'or' operator does not need to be called because the left hand side, the odd(), always succeeds. Again Lua knows that the conditional has succeeded and does not bother to evaluate the rest of the conditional.
Reporting these accurately is called 'branch coverage' and we can't do that for the same reasons we have difficulty when everything is crammed onto a single line.
#Testing and the pursuit of coverage
Coverage is not an end in itself, it is only an aid to evaluating the effectiveness of your unit tests. This may reveal parts of your code that can be changed but do not make changes to your code just to hit that magic 100% coverage. After a series of 'if' and 'elseif' statements you might put a final 'else' with an error that reads "This should never happen". This is called defensive programming and you may find that it seems to be impossible to actually trigger that condition. It's ok, better safe than sorry.
You can improve coverage in two ways: writing better tests (simply adding more tests does not necessarily improve coverage) or removing the offending code.
Never just remove code because testing it is too hard.