Guidelines for Testing Techniques and Test Measurement

B.13 Random Testing

**Introduction**

This
black box technique requires no partitioning of the component's input domain,
but simply requires test inputs to be chosen from this input domain at
random. We illustrate the technique by
means of an example.

**Example**

Consider
a component that transforms coordinates, with the following specification:

*The
component shall transform the Cartesian coordinates (x,y) for screen position
into their polar equivalent (r,**q**) using the equations*_{} and cos*q** = x/r. The origin
of the Cartesian coordinates and the pole of the polar coordinates shall be the
centre of the screen and the x-axis shall be considered the initial line for
the polar coordinates progressing counter-clockwise. All inputs and outputs shall be represented as fixed-point
numbers with both a range and a precision.
These shall be:*

__Inputs__

*x
- range -320..+320, in increments of 1/2*^{6}

*y
- range -240..+240, in increments of 1/2*^{7}

__Outputs__

*r
- range 0..400, in increments of 1/2*^{6}

*q** - range 0..(2**p**-1/2*^{6}), in increments of 1/2^{6}^{}

No
information is available about the operational distribution of the inputs to
the component, so a uniform distribution is used. For each test case a random test input value is selected for both
x and y, using the definitions provided in the component specification and
based on a uniform distribution across their defined ranges. From the definitions we can see that in any
one random test case x can take one of 40,961 values, while y can take one of
61,441 values. Care should be taken if
using an *expected* operational
distribution rather than a uniform distribution. An expected distribution that ignores parts of the input domain
can lead to unexpected error conditions being left untested.

Random
testing may be performed either manually or using automated test tools. Random testing is most cost-effective when
fully automated as then very many tests can be run without manual
intervention. However, to achieve full
automation it must be possible to:

· automatically
generate random test inputs; and

either · automatically
generate expected results from the specification;

or · automatically
check test outputs against the specification.

The
automatic generation of random test input values is not difficult using a
pseudo-random number generator as long as the component's inputs are
well-defined. If the test input values
are produced using a pseudo-random number generator, then these values do not
need to be recorded explicitly as the same set can be reproduced. This is normally possible if a 'seed' value
has been used to prime the pseudo-random number generator and this value is
recorded.

The
automatic generation of expected outputs or the automatic checking of outputs,
is however more problematic. Generally
it is not practicable to automatically generate expected outputs or
automatically check outputs against the specification, however for certain
components it is possible, such as where:

· trusted independently-produced software that
performs the same function as the component under test is available (presumably
not meeting the same constraints such as speed of processing, implementation language,
etc.);

· the test is concerned solely with whether the
component crashes or not (so the expected result is 'not to crash');

· the nature of the component's output makes it
relatively easy to check the result. An
example of this is a sort function where it is a simple task to automatically
check that the outputs have been sorted correctly;

· it is easy to generate inputs from the outputs
(using the inverse of the component's function). An example of this is a square root function where simply squaring
the output should produce the input.

The
coordinate transformation component can be checked automatically using the
inverse function approach. In this case, rcosq=x can be obtained directly from the
component's specification. By some
analysis rsinq=y can also be
deduced. If these two equations are
satisfied to a reasonable numerical tolerance then the component has
transformed the coordinates correctly.

Even
when full automation of random testing is not practicable its use should still
be considered as it does not carry the
large overhead of designing test cases as required by the non-random techniques.

While
no coverage measure is defined, a completion criterion can be set as a target
level of probability of non-conforming outcome (e.g. 0.001) together with
confidence interval bounds (say 95%).

For
components with larger input sets than this small example the "Symbolic Input
Attribute Decomposition" (SIAD) tree (ref: *Quality
Programming*, C-K Cho, 1987, ISBN 0-471-84899-9) is a useful method for organising
the input domain for random sampling before test case design.

B.14 Other Testing Techniques

Any techniques not explicitly defined
that are used should be sent to the editors to be considered for inclusion in a
later version of the Standard. These
details can be submitted in a variety of ways, as described in clause E.1