Test Technique Effectiveness
Up to this point the Standard has
provided no guidance on either the choice of test case design techniques or
test completion criteria (sometimes known as test adequacy criteria), other
than that they should be selected from clauses 3 and 4 respectively. The main reason for this is that there is no
established consensus on which techniques and criteria are the most
effective. The only consensus is that
the selection will vary as it should be dependent on a number of factors such
as criticality, application area, and cost.
Research into the relative effectiveness of test case design and
measurement techniques has, so far, produced no definitive results and although
some of the theoretical results are presented below it should be recognised
that they take no account of cost.
There is no requirement to choose
corresponding test case design and test measurement techniques. In fact it is good practice to use
functional test case design techniques and structural test measurement
techniques. Functional techniques are
effective at detecting errors of omission, while structural techniques can only
detect errors of commission. So a test
plan could typically require boundary value analysis to be used to generate an
initial set of test cases, while also requiring 100% branch coverage to be
achieved. This diverse approach could,
presumably, lead to branch testing being used to generate any supplementary
test cases required to achieve coverage of any branches missed by the boundary
value analysis test case suite.
Ideally the test coverage levels chosen
as test completion criteria should, wherever possible, be 100%. Strict definitions of test coverage levels
have sometimes made this level of coverage impracticable, however the
definitions in clause 4 have been defined to allow infeasible coverage items to
be discounted from the calculations thus making 100% coverage an achievable
goal.
With test completion criteria of 100%
(and only 100%) it is possible to
relate some of them in an ordering, where criteria are shown to subsume, or
include, other criteria. One criterion
is said to subsume another if, for all components and their specifications,
every test case suite that satisfies the first criterion also satisfies the
second. For example, branch coverage
subsumes statement coverage because if branch coverage is achieved (to 100%),
then statement coverage is always guaranteed to be achieved (to 100%) as well.
It should be noted that the 'subsumes'
relation described here strictly links test coverage criteria (rather than test
case design techniques) and so only provides an indirect indication of the
relative effectiveness of test case design techniques.
Not all test coverage criteria can be
related by the subsumes ordering and the functional and structural criteria are
not related at all. This leads to a
partial ordering of criteria, illustrated in figure C.1, where an arrow from
one criterion to another indicates that the first criterion subsumes the
second. Where a test coverage criterion
does not appear in the partial orderings then it is not related to any other
criterion by the subsumes relation.
Figure
C.1 Partial Ordering of Structural Test Coverage Criteria
Despite
its intuitive appeal the subsumes relation suffers a number of limitations that
should be considered before using it to choose test completion criteria:
. Firstly it relates only a subset of
the available test completion criteria and inclusion in this subset provides no
indication of effectiveness, so other criteria not shown in figure C.1 should
still be considered.
. Secondly, the subsumes relation
provides no measure of the amount by which one criterion subsumes another and
subsequently does not provide any measure of relative cost effectiveness.
. Thirdly, the partial orderings only
apply to single criteria while it is recommended that more than one criterion
is used, with at least one functional and one structural criterion.
. Finally, and most importantly, the
subsumes relation does not necessarily order test completion criteria in terms
of their ability to expose errors (their test effectiveness). It has been shown, for instance, that 100%
path coverage (when achievable) may not be as effective, for some components,
as some of the criteria it subsumes, such as those concerned with data
flow. This is because some errors are data
sensitive and will not be exposed by simply executing the path on which they
lie, but require variables to take a particular value as well (E.g. An
'unprotected' division by an integer variable may erroneously be included in a
component that will only fail if that variable takes a negative value). Satisfying data flow criteria can
concentrate the testing on these aspects of a component's behaviour, thus
increasing the probability of exposing such errors. It can be shown that in some circumstances test effectiveness is
increased by testing a subset of the paths required by a particular criteria
but exercising this subset with more test cases.
The subsumes relation is highly
dependent on the definition of full coverage for a criterion and although Figure
C.1 is correct for the definitions in clause 4 of this Standard it may not
apply to alternative definitions used elsewhere.
