Test case optimization techniques

Black Box Testing

Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would only know the "legal" inputs and what the expected outputs should be, but not how the program actually arrives at those outputs. It is because of this that black box testing can be considered testing with respect to the specifications, no other knowledge of the program is necessary

Black - Box Testing

Black Box Testing:

Focus on functional requirements.

Compliments white box testing.

“A programmer makes mistakes by Omission as well as Commission”

Black box testing is concerned only with testing the specification, it cannot guarantee that all parts of the implementation have been tested. Thus black box testing is testing against the specification and will discover faults of omission, indicating that part of the specification has not been fulfilled. White box testing is testing against the implementation and will discover faults of commission, indicating that part of the implementation is faulty. In order to fully test a software product both black and white box testing are required.

Black Box Testing Attempts to find:

1.      incorrect or missing functions

2.      interface errors

3.      errors in data structures or external database access

4.      performance errors

5.      initialization and termination errors

The advantages of this type of testing include:

·         The test is unbiased because the designer and the tester are independent of each other.

·         The tester does not need knowledge of any specific programming languages.

·         The test is done from the point of view of the user, not the designer.

·         Test cases can be designed as soon as the specifications are complete.

The disadvantages of this type of testing include:

·         The test can be redundant if the software designer has already run a test case.

·         The test cases are difficult to design.

·         Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.

Black Box Testing Methods:

1.      Equivalence Partitioning

2.      Boundary Value Analysis.

3.      Cause Effect Graphing Techniques.

4.      Comparison Testing

Equivalence Partitioning

Equivalence partitioning is a black-box testing method that divides the input domain of a program into classes of data from which test cases can be derived.

An ideal test case single-handedly uncovers a class of errors that might otherwise require many cases to be executed before the general error is observed. Equivalence partitioning strives to define a test case that uncovers classes of errors, thereby reducing the total number of test cases that must be developed.

– An equivalence class represents a set of valid or invalid classes for an input condition

–        An input condition is a specific numeric value, a range of values, a set of related values or a Boolean condition (true or false)

Equivalence Partitioning

• Example: Suppose a program called for an input that is a 5-digit integer between 10,000 and 99,999. This input condition defines the following equivalence classes:

–        equivalence partitions are <10,000, 10,000-99,999 and >10, 000

–        Test cases would be selected from each equivalence class i.e. 00000, 09999, 10000, 99999, 10001

If input condition specifies

– A range, 1valid and 2 invalid classes are defined

– A specific value, 1 valid and 2 invalid classes are defined

– A member of a set, 1 valid and 1 invalid classes are defined

–        A Boolean condition, 1 valid and 1 invalid classes are defined

Basic idea:

Partition the input domain of a program into a finite number of equivalence classes such that one can reasonably assume that a test of a representative value of each class is equivalent to a test of any other value.

The method:

Step 1: Study the program specification to find input conditions and identify the equivalence classes.

(1) for each condition Pi, identify one valid equivalence class defined by Pi and one

invalid equivalence class defined by ¬Pi. e.g., if Pi is "the first symbol of the identifier must be a letter" then the valid equivalence class is defined by "it is a letter" and the invalid one is defined by "it is not a letter".

(2) for each input condition of the form Pi and Pj, identify one valid equivalence class

defined by Pi and Pj, and two invalid ones defined by ¬ Pi and ¬ Pj, respectively.

e.g., if the input condition is 1 = n = 5, the valid equivalence class is defined by 1 = n = 5, and the two invalid equivalence classes are defined by n < 1 and n > 5, respectively.

(3) if there is any reason to suspect that elements in an equivalence class are not treated in an identical manner by the program, split the equivalence class into smaller ones.

The method:

Step 2: Select the test cases.

(1) Until all valid equivalence classes have been covered by test cases, find a new test case that covers as many of the uncovered valid equivalence classes as possible.

(2) For every invalid equivalence class, find a test case that covers that invalid equivalence class only.

Boundary Value Analysis

Since a greater number of errors tend to occur at the boundaries of the input domain than in the "center", boundary value analysis has been developed as a testing technique. BVA compliments equivalence partitioning and leads to a selection of test cases that exercise bounding values. Rather than focusing solely on input conditions, BVA derives test cases from the output domain as well.

• Boundary Value Analysis is a black box testing technique that where test cases are designed to test the boundary of an input domain. Studies have shown that more errors occur on the "boundary" of an input domain rather than on the "center".

• Boundary value analysis complements and can be used in conjunction with

equivalence partitioning.

– Test at min and max values of an input and output range, and just below max and just above min values

– Test values at input and output min and max numbers and just above min and just below max numbers

– Test at data structure boundaries

Basic idea:

Select test cases that lie directly on, above, and beneath the boundaries of input  equivalence classes and output equivalence classes to explore the program behavior along the border.

(The counterpart of domain-strategy testing)

Boundary-value analysis

This method differs from the equivalence partitioning in two respects:

(a) rather than checking to see if the program will execute correctly for a representative element in an equivalence class, it attempts to determine if the program defines the equivalence class correctly, and

(b) rather than selecting test cases based on input conditions only, it also requires derivation of test cases base on output conditions.

The method:

(1) If an input variable is defined in a range from LB to UB, use LB, UB, LB - ? , and UB + ? as the test cases. (Here delta represents the smallest possible change in value.)

(2) Use rule (1) for each output variable.

(3) If the input or output of a program is a sequence (e.g., a sequential file or a linear list), focus attention on the first and last element of the sequence.

(4) Use your ingenuity to search for additional boundary values.

Cause Effect Graphing Techniques

Translation of natural language descriptions of procedures to software based algorithms is error prone.

–        Causes and effects are listed, graphed, converted to a decision table, and then to test cases

Basic idea:

A "cause" is an input condition, and an "effect" is a specific sequence of computations to be performed. A cause-effect graph is basically a directed graph that describes the logical combinations of causes and their relationship to the effects to be produced.

Cause-effect graphing is a technique that aids in selecting test cases to check if the program will produce right effect for every possible combination of causes.

The method:

(1) Divide the program specification into pieces of workable size.

(2) Identify causes and effects in the specification.

(3) Analyze the specification to determine the logical relationship among causes and effects, and express it as a cause effect graph.

(4) Identify syntactic or environmental constraints that make certain combinations impossible.

(5) Translate the graph into a limited-entry decision table.

(6) Select a test case for every column in the decision table.

The format of a limited-entry decision table

condition stubs	condition entries
action stubs	action entries

• The condition stub contains a list of conditions, one to each row.

• The condition entry lists combinations of condition values in a column.

• The condition constraints express interactions between conditions.

• The action stub contains a list of actions.

• The action entry contains an "X" for each action to be performed.

• The application constraints express the conditions under which an action is to be performed.

Example:

Shown below is an example of a limited-entry decision table obtained by analyzing and completing the following program specification:

From US Army Corps of Engineers:

Executive Order 10358 provides in the case of an employee whose work week varies from the normal Monday through Friday work week, that Labor Day and Thanksgiving Day each were to be observed on the next succeeding workday when the holiday fell on a day outside the employee's regular basic work week. Now, when Labor Day, Thanksgiving Day or any of the new Monday holidays are outside an employee's basic workbook, the immediately preceding workday will be his holiday when the non-workday on which the holiday falls is the second non-workday or the non-workday designated as the employee's day off in lieu of Saturday. When the non-workday on which the holiday falls is the first non-workday or the non-workday designated as the employee's day off in lieu of Sunday, the holiday observance is moved to the next succeeding workday.

How do you test code which attempts to implement this?

Cause-effect graphing attempts to provide a concise representation of logical combinations and corresponding actions.

1.      Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned to each.

2.      A cause-effect graph developed.

3.      Graph converted to a decision table.

4.      Decision table rules are converted to test cases.

Comparison Testing

In some applications the reliability is critical.

Redundant hardware and software may be used.

For redundant s/w, use separate teams to develop independent versions of the software.

Test each version with same test data to ensure all provide identical output.

Run all versions in parallel with a real-time comparison of results.

Even if will only run one version in final system, for some critical applications can develop independent versions and use comparison testing or back-to-back testing.

When outputs of versions differ, each is investigated to determine if there is a defect.

Method does not catch errors in the specification.