BS7925-2 Annex B - Guidelines for Testing and Test Measurement

Guidelines for Testing Techniques and Test Measurement

B.13  Random Testing

Introduction

This black box technique requires no partitioning of the component's input domain, but simply requires test inputs to be chosen from this input domain at random.  We illustrate the technique by means of an example.

 

Example

Consider a component that transforms coordinates, with the following specification:

 

The component shall transform the Cartesian coordinates (x,y) for screen position into their polar equivalent (r,q) using the equations and cosq = x/r.  The origin of the Cartesian coordinates and the pole of the polar coordinates shall be the centre of the screen and the x-axis shall be considered the initial line for the polar coordinates progressing counter-clockwise.  All inputs and outputs shall be represented as fixed-point numbers with both a range and a precision.  These shall be:

Inputs

x - range -320..+320, in increments of 1/2⁶

y - range -240..+240, in increments of 1/2⁷

Outputs

r - range 0..400, in increments of 1/2⁶

q - range 0..(2p-1/2⁶), in increments of 1/2⁶

 

No information is available about the operational distribution of the inputs to the component, so a uniform distribution is used.  For each test case a random test input value is selected for both x and y, using the definitions provided in the component specification and based on a uniform distribution across their defined ranges.  From the definitions we can see that in any one random test case x can take one of 40,961 values, while y can take one of 61,441 values.  Care should be taken if using an expected operational distribution rather than a uniform distribution.  An expected distribution that ignores parts of the input domain can lead to unexpected error conditions being left untested.

Random testing may be performed either manually or using automated test tools.  Random testing is most cost-effective when fully automated as then very many tests can be run without manual intervention.  However, to achieve full automation it must be possible to:

         ·     automatically generate random test inputs; and

either ·     automatically generate expected results from the specification;

or      ·     automatically check test outputs against the specification.

 

The automatic generation of random test input values is not difficult using a pseudo-random number generator as long as the component's inputs are well-defined.  If the test input values are produced using a pseudo-random number generator, then these values do not need to be recorded explicitly as the same set can be reproduced.   This is normally possible if a 'seed' value has been used to prime the pseudo-random number generator and this value is recorded.

The automatic generation of expected outputs or the automatic checking of outputs, is however more problematic.  Generally it is not practicable to automatically generate expected outputs or automatically check outputs against the specification, however for certain components it is possible, such as where:

·  trusted independently-produced software that performs the same function as the component under test is available (presumably not meeting the same constraints such as speed of processing, implementation language, etc.);

·  the test is concerned solely with whether the component crashes or not (so the expected result is 'not to crash');

·  the nature of the component's output makes it relatively easy to check the result.  An example of this is a sort function where it is a simple task to automatically check that the outputs have been sorted correctly;

·  it is easy to generate inputs from the outputs (using the inverse of the component's function).  An example of this is a square root function where simply squaring the output should produce the input.

 

The coordinate transformation component can be checked automatically using the inverse function approach. In this case, rcosq=x can be obtained directly from the component's specification.  By some analysis rsinq=y can also be deduced.  If these two equations are satisfied to a reasonable numerical tolerance then the component has transformed the coordinates correctly.

Even when full automation of random testing is not practicable its use should still be considered as  it does not carry the large overhead of designing test cases as required by the non-random techniques.

While no coverage measure is defined, a completion criterion can be set as a target level of probability of non-conforming outcome (e.g. 0.001) together with confidence  interval bounds (say 95%).

For components with larger input sets than this small example the "Symbolic Input Attribute Decomposition" (SIAD) tree (ref: Quality Programming, C-K Cho, 1987, ISBN 0-471-84899-9) is a useful method for organising the input domain for random sampling before test case design.

 

B.14  Other Testing Techniques

Any techniques not explicitly defined that are used should be sent to the editors to be considered for inclusion in a later version of the Standard.  These details can be submitted in a variety of ways, as described in clause E.1