The Test Management Guide Software Test Guide RSS Feed


BS7925-2 Annex C - Test Technique Effectiveness

Annex C   

Test Technique Effectiveness

Up to this point the Standard has provided no guidance on either the choice of test case design techniques or test completion criteria (sometimes known as test adequacy criteria), other than that they should be selected from clauses 3 and 4 respectively.  The main reason for this is that there is no established consensus on which techniques and criteria are the most effective.  The only consensus is that the selection will vary as it should be dependent on a number of factors such as criticality, application area, and cost.  Research into the relative effectiveness of test case design and measurement techniques has, so far, produced no definitive results and although some of the theoretical results are presented below it should be recognised that they take no account of cost.

There is no requirement to choose corresponding test case design and test measurement techniques.  In fact it is good practice to use functional test case design techniques and structural test measurement techniques.  Functional techniques are effective at detecting errors of omission, while structural techniques can only detect errors of commission.  So a test plan could typically require boundary value analysis to be used to generate an initial set of test cases, while also requiring 100% branch coverage to be achieved.  This diverse approach could, presumably, lead to branch testing being used to generate any supplementary test cases required to achieve coverage of any branches missed by the boundary value analysis test case suite.

Ideally the test coverage levels chosen as test completion criteria should, wherever possible, be 100%.  Strict definitions of test coverage levels have sometimes made this level of coverage impracticable, however the definitions in clause 4 have been defined to allow infeasible coverage items to be discounted from the calculations thus making 100% coverage an achievable goal.

With test completion criteria of 100% (and only 100%) it is possible to relate some of them in an ordering, where criteria are shown to subsume, or include, other criteria.  One criterion is said to subsume another if, for all components and their specifications, every test case suite that satisfies the first criterion also satisfies the second.  For example, branch coverage subsumes statement coverage because if branch coverage is achieved (to 100%), then statement coverage is always guaranteed to be achieved (to 100%) as well.

It should be noted that the 'subsumes' relation described here strictly links test coverage criteria (rather than test case design techniques) and so only provides an indirect indication of the relative effectiveness of test case design techniques.

Not all test coverage criteria can be related by the subsumes ordering and the functional and structural criteria are not related at all.  This leads to a partial ordering of criteria, illustrated in figure C.1, where an arrow from one criterion to another indicates that the first criterion subsumes the second.  Where a test coverage criterion does not appear in the partial orderings then it is not related to any other criterion by the subsumes relation.



Figure C.1 Partial Ordering of Structural Test Coverage Criteria



Despite its intuitive appeal the subsumes relation suffers a number of limitations that should be considered before using it to choose test completion criteria:

.           Firstly it relates only a subset of the available test completion criteria and inclusion in this subset provides no indication of effectiveness, so other criteria not shown in figure C.1 should still be considered.

.           Secondly, the subsumes relation provides no measure of the amount by which one criterion subsumes another and subsequently does not provide any measure of relative cost effectiveness.

.           Thirdly, the partial orderings only apply to single criteria while it is recommended that more than one criterion is used, with at least one functional and one structural criterion.

.           Finally, and most importantly, the subsumes relation does not necessarily order test completion criteria in terms of their ability to expose errors (their test effectiveness).  It has been shown, for instance, that 100% path coverage (when achievable) may not be as effective, for some components, as some of the criteria it subsumes, such as those concerned with data flow.  This is because some errors are data sensitive and will not be exposed by simply executing the path on which they lie, but require variables to take a particular value as well (E.g. An 'unprotected' division by an integer variable may erroneously be included in a component that will only fail if that variable takes a negative value).  Satisfying data flow criteria can concentrate the testing on these aspects of a component's behaviour, thus increasing the probability of exposing such errors.  It can be shown that in some circumstances test effectiveness is increased by testing a subset of the paths required by a particular criteria but exercising this subset with more test cases.

The subsumes relation is highly dependent on the definition of full coverage for a criterion and although Figure C.1 is correct for the definitions in clause 4 of this Standard it may not apply to alternative definitions used elsewhere.


© RuleWorks - All Rights Reserved - Policy - - Sitemap