Posts Tagged Software Coverage

Misapprehension 13: Branch coverage subsumes statement coverage, therefore you can conclude 50% statement coverage from 50% branch coverage

Branch coverage actually subsumes statement coverage, i.e. you can conclude 100% statement coverage from 100% branch coverage. But you cannot conclude 50% statement coverage from 50% branch coverage. See the figure below for an example.

Can you conclude 50% statement coverage from 50% branch coverage?

Can you conclude 50% statement coverage from 50% branch coverage?

If you execute an arbitrary test case for the code snippet in the figure above, you will get 50% branch coverage, but you will not get 50% statement coverage, because the two branches that are present contain a different number of statements each.

, , ,

No Comments

Misapprehension 12: With respect to coverage, the programming language is irrelevant

Is the number of test cases needed for a certain coverage measure dependent from the programming language or not? Let’s consider the decision “A and B” in both the Pascal programming language and in the C programming language.

Are Pascal and C equivalent with respect to coverage?

Are Pascal and C equivalent with respect to coverage?

The figure above lists all the possible test cases in the particular programming languages. To get 100% simple condition coverage for instance, you need only two test cases in the case of Pascal (i.e. no. 2 and no. 3), whereas in case of C you need three test cases (no. I, II, and III). This is because Pascal is a programming language featuring complete evaluation, and C is not.

, , ,

No Comments

Misapprehension 11: With respect to coverage, it makes no difference how a compound decision is written

Are these two decisions equivalent with respect to coverage?

Are these two decisions equivalent with respect to coverage?

The two compound decisions D1 and D2 in the figure above are functionally equivalent, i.e. all possible input combinations yield the same result. However, in the C/C++ programming language, there are differences with respect to coverage. For instance, to reach 100% Multiple Condition Coverage (MCC) for D1, you need five test cases, whereas for D2, you only need four. This is caused by the incomplete evaluation (aka short-circuit operation) of the C/C++ programming language. As soon as the compiler knows the outcome of the whole decision, evaluation is stopped. Therefore, for D2 only four test cases are possible, which are (or need to be) sufficient for 100% MC/DC.

[BTW: Because the C/C++ compiler always evaluates decisions from left to right, usage of D2 will lead to faster evaluation on average, because the short-circuit operation will occur more often for D2 than for D1.]

, , ,

No Comments

Misapprehension 10: We must reach 100% MC/DC, because then our software is error-free

It is correct, that Modified Condition / Decision Coverage (MC/DC) is one of the more sophisticated coverage measures, which for instance needs to be provided for products on level A (the most safety-critical level) according to DO-178B. However, to reach 100% MC/DC you usually need more test cases (i.e. test effort) than for less demanding measures, e.g. branch coverage. Is this higher testing effort worthwhile? If you have a huge budget for testing, you certainly should consider aiming for 100% MC/DC. Unfortunately, in reality, the budget for testing is often limited. Therefore, you should ask yourself, if you shouldn’t invest the additional effort required to reach 100% MC/DC in other kinds of tests, e.g. in reviews, mutation testing, or static analysis. The rationale for this is that even 100% MC/DC does not guarantee error-free code, because e.g. MC/DC is insensitive against errors in calculations, like any other coverage measure.
However, the author must admit, that he knows from a case in real life, where unit testing was already finished with 100% branch coverage reached. Then, the requirement for 100% MC/DC was raised, which required additional test cases, which actually revealed a programming error. This error would have gone undetected without the objective for 100% MC/DC.

, , , ,

No Comments

Misapprehension 8: Code Coverage measures the quality of your test cases

Not always. Let’s take as an example a function that calculates the sinus function:

Is 100% coverage enough for that function?

Is 100% coverage enough for that function?

For the function in the figure above, a single test case, e.g. with an input value of 0 for the variable x_deg is sufficient to reach 100% branch/decision coverage, 100% MC/DC, and 100% MCC. The actual result is 0, what is the expected and correct value. But this single test case neither is error-sensitive nor is a single test case sufficient, even if you reach 100% coverage.

, , ,

No Comments

Advices

If the topic is software coverage (or code coverage): Please make sure that you and your counterpart understand the same measure by the same term.

If you use a tool to determine software coverage: Please check how the tool handles the two cases of doubt given in misapprehension 4.

Please keep an eye on the effort you need to reach e.g. 100% MC/DC: Is it in the right relation to the gain in relevance? Sometimes it may be better to spend this effort for other test activities!

Beware of fool’s gold: Don’t relax because you have reached 100%!

, , ,

No Comments

Home

The Most Common Misapprehensions about Software Coverage

More and more products need a certification according to standards like IEC 61508 or DO-178B. This in turn requires measuring the coverage of the software in these products during testing. However, a lot of misapprehensions and factoids about software coverage are widely spread. In the following, some of these shall be clarified.
Software coverage is also known as code coverage.

Please note: This is not an introduction to software coverage or code coverage. Such an introduction can be found here.

Read on:


, , , , ,

No Comments