Posts Tagged Code Coverage

Misapprehension 13: Branch coverage subsumes statement coverage, therefore you can conclude 50% statement coverage from 50% branch coverage

Branch coverage actually subsumes statement coverage, i.e. you can conclude 100% statement coverage from 100% branch coverage. But you cannot conclude 50% statement coverage from 50% branch coverage. See the figure below for an example.

Can you conclude 50% statement coverage from 50% branch coverage?

Can you conclude 50% statement coverage from 50% branch coverage?

If you execute an arbitrary test case for the code snippet in the figure above, you will get 50% branch coverage, but you will not get 50% statement coverage, because the two branches that are present contain a different number of statements each.

, , ,

No Comments

Misapprehension 12: With respect to coverage, the programming language is irrelevant

Is the number of test cases needed for a certain coverage measure dependent from the programming language or not? Let’s consider the decision “A and B” in both the Pascal programming language and in the C programming language.

Are Pascal and C equivalent with respect to coverage?

Are Pascal and C equivalent with respect to coverage?

The figure above lists all the possible test cases in the particular programming languages. To get 100% simple condition coverage for instance, you need only two test cases in the case of Pascal (i.e. no. 2 and no. 3), whereas in case of C you need three test cases (no. I, II, and III). This is because Pascal is a programming language featuring complete evaluation, and C is not.

, , ,

No Comments

Misapprehension 11: With respect to coverage, it makes no difference how a compound decision is written

Are these two decisions equivalent with respect to coverage?

Are these two decisions equivalent with respect to coverage?

The two compound decisions D1 and D2 in the figure above are functionally equivalent, i.e. all possible input combinations yield the same result. However, in the C/C++ programming language, there are differences with respect to coverage. For instance, to reach 100% Multiple Condition Coverage (MCC) for D1, you need five test cases, whereas for D2, you only need four. This is caused by the incomplete evaluation (aka short-circuit operation) of the C/C++ programming language. As soon as the compiler knows the outcome of the whole decision, evaluation is stopped. Therefore, for D2 only four test cases are possible, which are (or need to be) sufficient for 100% MC/DC.

[BTW: Because the C/C++ compiler always evaluates decisions from left to right, usage of D2 will lead to faster evaluation on average, because the short-circuit operation will occur more often for D2 than for D1.]

, , ,

No Comments

Misapprehension 10: We must reach 100% MC/DC, because then our software is error-free

It is correct, that Modified Condition / Decision Coverage (MC/DC) is one of the more sophisticated coverage measures, which for instance needs to be provided for products on level A (the most safety-critical level) according to DO-178B. However, to reach 100% MC/DC you usually need more test cases (i.e. test effort) than for less demanding measures, e.g. branch coverage. Is this higher testing effort worthwhile? If you have a huge budget for testing, you certainly should consider aiming for 100% MC/DC. Unfortunately, in reality, the budget for testing is often limited. Therefore, you should ask yourself, if you shouldn’t invest the additional effort required to reach 100% MC/DC in other kinds of tests, e.g. in reviews, mutation testing, or static analysis. The rationale for this is that even 100% MC/DC does not guarantee error-free code, because e.g. MC/DC is insensitive against errors in calculations, like any other coverage measure.
However, the author must admit, that he knows from a case in real life, where unit testing was already finished with 100% branch coverage reached. Then, the requirement for 100% MC/DC was raised, which required additional test cases, which actually revealed a programming error. This error would have gone undetected without the objective for 100% MC/DC.

, , , ,

No Comments

Misapprehension 9: It is sufficient to derive the test cases for 100% coverage from the code

Certainly it is possible to derive test cases from the existing source code, with the aim to get 100% coverage by execution of these test cases. There are even tools that do that automatically for you. But you need to be aware of two things thereby:
(1) You cannot detect omissions! From test cases that were derived from the code you will never find out that a part of the code is missing. Think of a check for a maximal value that was forgotten to be implemented. It is simply not possible to derive a test case from the code that will reveal that a piece of the code is not there. If you postulate your code to be complete, the test statement is doubtful, even if you have reached 100% coverage.

(2) The actual result of automatically derived test cases needs to be evaluated against the expected result. If you neglect this you will detect only fatal problems of your test object, e.g. a crash of your test object. If you simply accept the actual results from the execution of the given code, the relevance of the test is strongly reduced. Let’s take as an example for an undetected omission the sinus function from the precedent misapprehension. We ran a test case and got 100%. But we did not detect that the code actually missed some lines.

The omission (in red) was not detected by measuring code coverage

The omission (in red) was not detected by measuring code coverage

Excursus: What is the rationale of the added code? The added code makes the calculation more accurate, because, due to the fixed number of iterations of the for-loop, the result of the calculation becomes more inaccurate the higher the input value is. This is remedied by the added code, which makes use of the fact that sin(x°+360°) = sin(x°). It could well be the case that the specification for the sinus function states to pre-process the input value to get a value below 360° in any case, but the developer of the function has forgotten to implement the necessary code. The function works in spite of this, but not as accurate as it should.
An again: If we would run some test cases on the code in the figure above, and if we would achieve 100% coverage in doing so: Is the code in the figure above complete? I guess not: Consider input values below -360°, e.g. -400°.

, ,

No Comments

Misapprehension 8: Code Coverage measures the quality of your test cases

Not always. Let’s take as an example a function that calculates the sinus function:

Is 100% coverage enough for that function?

Is 100% coverage enough for that function?

For the function in the figure above, a single test case, e.g. with an input value of 0 for the variable x_deg is sufficient to reach 100% branch/decision coverage, 100% MC/DC, and 100% MCC. The actual result is 0, what is the expected and correct value. But this single test case neither is error-sensitive nor is a single test case sufficient, even if you reach 100% coverage.

, , ,

No Comments

Misapprehension 7: Code Coverage measures the quality of your code

Not absolutely. You only can say: The higher the percentage of coverage is, the “better” the code is tested with respect to that coverage measure. You might think this splits a too fine hair, but 100% coverage does not mean 100% software quality.

, ,

No Comments

Advices

If the topic is software coverage (or code coverage): Please make sure that you and your counterpart understand the same measure by the same term.

If you use a tool to determine software coverage: Please check how the tool handles the two cases of doubt given in misapprehension 4.

Please keep an eye on the effort you need to reach e.g. 100% MC/DC: Is it in the right relation to the gain in relevance? Sometimes it may be better to spend this effort for other test activities!

Beware of fool’s gold: Don’t relax because you have reached 100%!

, , ,

No Comments

Home

The Most Common Misapprehensions about Software Coverage

More and more products need a certification according to standards like IEC 61508 or DO-178B. This in turn requires measuring the coverage of the software in these products during testing. However, a lot of misapprehensions and factoids about software coverage are widely spread. In the following, some of these shall be clarified.
Software coverage is also known as code coverage.

Please note: This is not an introduction to software coverage or code coverage. Such an introduction can be found here.

Read on:


, , , , ,

No Comments