Testing and Debugging

Topics to be covered
1. Introduction to Testing
2. Levels of Testing
3. Test Techniques
4. when to stop testing
5. Debugging

Introduction to Testing

1.1 A Self Assessment Test

1.2 Correctness of a software

1.3 Testing Approaches

Testing

Testing is an important, mandatory part of
software development;
it is a technique for evaluating product quality
and also for indirectly improving it, by
identifying defects and problems

Correctness of a software

How do we decide correctness of a
software?
To answer this question, we need to first
understand how a software can fail.
Failure may be due to:
Error or incompleteness in the requirements
Difficulty in implementing the specification in
the target environment
Faulty system or program design
Defects in the code

Importance of testing
it is clear that testing cannot be seen as an
activity that will start after coding phase
Software testing is an activity that
encompasses the whole development life
cycle.
We need to test a program to demonstrate
the existence of a fault.
Contradictory to the terms, demonstrating
correctness is not a demonstration that the
program works properly.

The Art of Software Testing
principles :
Definition: Testing is the process of executing
a program with the intent of finding errors
A good test case is one that has a high
probability of detecting an as-yet
undiscovered error
A successful test case is one that detects an
as-yet undiscovered error.


more testing principles
Test case definition includes expected output
Programmers should avoid testing their own
programs (third party testing?)
Inspect the result of each test
Include test cases for invalid & unexpected input
conditions
Test cases serve as a documentation for future
maintenance
Do not assume that the program is bug free
The probability of more errors is proportional to the
number of errors already found

Example
If we abstract a software to be a
mathematical function f, which is expected to
produce various outputs for inputs from a
domain
It is clear from the definition of
Testing by Myers that
Testing cannot prove correctness of a
program it is just a series of experiments to
find out errors
There is nothing like 100% error-free code as
it is not feasible to conduct exhaustive testing,
proving 100% correctness of a program is not
possible
Why testing?
egoless programming and keep a goal of
eliminating as many faults as possible.

We need to place static testing also in place
to capture an error before it becomes a defect
in the software.
1.3 Testing Approaches
There are two major approaches to testing:
Black-box (or closed box or data-driven or
input/output driven or behavioral) testing
White-box (or clear box or glass box or logic-
driven) testing

black-box
If the testing component is viewed as a black-box, the
inputs have to be given to observe the behavior (output)
of the program.
In this case, it makes sense to give both valid and
invalid inputs.
The observed output is then matched with expected
result (from specifications).
The advantage of this approach is that the tester need
not worry about the internal structure of the program.
However, it is impossible to find all errors using this
approach.
For instance, if one tried three equilateral triangle test
cases for the triangle program, we cannot be sure
whether the program will detect all equilateral triangles.
The program may contain a hard coded display of
scalene triangle for values (300,300,300).
To exhaustively test the triangle program, we need to
create test cases for all valid triangles up to MAXIMUM
integer size.
To be sure of finding all possible errors, we not only test
with valid inputs but all possible inputs (including invalid
ones like characters, float, negative integers, etc).

White-box
This approach examines the internal structure of the
program.
This means logical analysis of software element
where testing is done to trace all possible paths of
control flow.
This method of testing exposes both errors of
omission (errors due to neglected specification) and
also errors of commission (something not defined
by the specification).
Let us look at a specification of a simple file
handling program

The program has to read employee details such as
employee id, name, date of joining and department as
input from the user, create a file of employees and
display the records sorted in the order of employee ids.
Examples for errors of omission: Omission of Display
module, Display of records not in sorted order of
employee ids, file created with fewer fields etc.
Example of error of commission: Additional lines of code
deleting some arbitrary records from the created file. (No
amount of black box testing can expose such errors of
commission as the method uses specification as the
reference to prepare test cases).
However, it is many a time practically impossible to do
complete white box testing to trace all possible paths of
control flow as the number of paths could astronomically
large.
For instance, a program segment which has 5 different
control paths (4 nested if-then-else) and if this segment
is iterated 20 times.
The number of unique paths would be 5
20
+5
19
++5
1
=
10
14
or 100 trillion.
If we were able to complete one test case every 5
minutes, it would take approximately one billion years to
test every unique path.
Due to dependency of decisions, not all control
paths may be feasible.
Hence, actually, we may not be testing all these
paths.
Exhaustive path testing will not address missing
paths and data-sensitive errors.
A black-box approach would capture these
errors!
conclusion
black-box & White-box
It is not feasible to do exhaustive testing
either in block or in white box approaches.
None of these approaches are superior
meaning, one has to use both approaches, as
they really complement each other.
Not, but not the least, static testing still plays
a large role in software testing
2. Levels of Testing
In developing a large system, testing usually
involves several stages.
Unit Testing
Integration Testing
System Testing
Acceptance Testing


Unit Testing
Initially, each program component (module) is tested
independently verifying the component functions with
the types of input identified by studying components
design.

Unit testing is done in a controlled environment with a
predetermined set of data fed into the component to
observe what output actions and data are produced.

Integration Testing

When collections of components have been
unit-tested, the next step is ensuring that the
interfaces among the components are
defined and handled properly.
This process of verifying the synergy of
system components against the program
Design Specification is called Integration
Testing.
System Testing
Once the system is integrated, the overall
functionality is tested against the Software
Requirements Specification (SRS).
Then, the other non-functional requirements
like performance testing are done to ensure
readiness of the system to work successfully
in a customers actual working environment.
This step is called System Testing.
Acceptance Testing
The next step is customers validation of the
system against User Requirements
Specification (URS).
Customer in their working environment does
this exercise of Acceptance Testing usually
with assistance from the developers. Once
the system is accepted, it will be installed and
will be put to use.
2.2 Unit Testing
Examining the code: Typically the static
testing methods like:
Reviews, Walkthroughs and Inspections are
used.
Proving code correct
Use formal methods.
One way to investigate program correctness
is to view the code as a statement of logical
flow. Using mathematical logic, if we can
formulate the program as a set of assertions
and theorems, we can show that the truth of
the theorems implies the correctness of the
code.
Use of this approach forces us to be more
rigorous and precise in specification.
Much work is involved in setting up and
carrying out the proof.
For example, the code for performing bubble
sort is much smaller than its logical description
and proof.


Testing program components
In the absence of simpler methods and
automated tools, Proving code correctness
will be an elusive goal for software engineers.
Proving views programs in terms of classes of
data and conditions and the proof may not
involve execution of the code.
While proof tells us how a program will work
in a hypothetical environment described by
the design and requirements, testing gives us
information about how a program works in its
actual operating environment.

To test a component (module), input data and
conditions are chosen to demonstrate an
observable behavior of the code.
A test case is a particular choice of input data
to be used in testing a program.
Test case are generated by using either
black-box or white-box approaches
2.3 Integration Testing
Integration is the process of assembling unit-tested
modules.
We need to test the following aspects:
Interfaces: To ensure interface integrity, the transfer of
data between modules is tested. When data is passed to
another module, by way of a call, there should not be any
loss or corruption of data. The loss or corruption of data can
happen due to miss-match or differences in the number or
order of calling and receiving parameters.
Module combinations may produce a different behavior due
to combinations of data that are not exercised during unit
testing.
Global data structures, if used, may reveal errors due to
unintended usage in some module.

Integration Strategies

Depending on design approach, one of the
following integration strategies can be
adopted:
Big Bang approach
Incremental approach
Top-down testing
Bottom-up testing
Sandwich testing

consider the following arrangement of
modules:
Big Bang approach
Though Big Bang approach seems to be
advantageous when we construct
independent module concurrently,
this approach is quite challenging and risky
as we integrate all modules in a single step
and test the resulting system.
Locating interface errors, if any, becomes
difficult here.
Incremental approach,
The alternative strategy is an incremental
approach, wherein modules of a system are
consolidated with already tested components of
the system.
In this way, the software is gradually built up,
spreading the integration testing load more
evenly through the construction phase.
Incremental approach can be implemented in two
distinct ways:
Top-down
Bottom-up.
Top-down
Bottom-up testing

TOP-DOWN Vs BOTTOM-UP
sandwich testing
To overcome the limitations and to exploit the
advantages of Top-down and Bottom-up testing, a
sandwich testing is used
This system is viewed as three layers the target
layer in the middle, the levels above the target, and
the levels below the target.
A top-down approach is used in the top layer and a
bottom-up one in the lower layer.


Testing converges on the target layer, chosen
on the basis of system characteristics and the
structure of the code.
For example, if the bottom layer contains
many general-purpose utility programs, the
target layer (the one above) will be
components using the utilities.
This approach allows bottom-up testing to
verify the utilities correctness at the
beginning of testing.
Choosing an integration strategy depends not
only on system characteristics, but also on
customer expectations.
For instance, the customer may want to see a
working version as soon as possible, so we
may adopt an integration schedule that
produces a basic working system early in the
testing process.
In this way coding and testing can go
concurrently.
2.4 System Testing

The objective of unit and integration testing
was to ensure that the code implemented the
design properly.
In system testing, we need to ensure that the
system does what the customer wants it to
do. Initially the functions (functional
requirements) performed by the system are
tested.
A function test checks whether the integrated
system performs its functions as specified in
the requirements.

After ensuring that the system performs the
intended functions, the performance test is
done.
This non-functional requirement includes
security, accuracy, speed, and reliability.

2.5 Acceptance Testing
Acceptance testing is the customer (and user)
evaluation of the system, primarily to
determine whether the system meets their
needs and expectations.
Usually acceptance test is done by customer
with assistance from developers.
Customers can evaluate the system either by
conducting a benchmark test or by a pilot
test.
In benchmark test, the system performance
is evaluated against test cases that represent
typical conditions under which the system will
operate when actually installed.
A pilot test installs the system on an
experimental basis, and the system is
evaluated against everyday working.

alpha test & beta test
Sometimes the system is piloted in-house
before the customer runs the real pilot test.
The in-house test, in such case, is called an
alpha test,
and the customers pilot is a beta test. This
approach is common in the case of
commercial software where the system has to
be released to a wide variety of customers.
parallel testing
This is a third approach
is done when a new system is replacing an
existing one or is part of a phased
development.
The new system is put to use in parallel with
previous version and will facilitate gradual
transition of users, and to compare and
contrast the new system with the old.
3. Test Techniques
Black box testing
White box testing
Black box testing

Black Box Approaches:
- Equivalence Partitioning
- Boundary Value Analysis
- Cause Effect Analysis
- Cause Effect Graphing
- Error Guessing
Equivalence Partitioning
Equivalence partitioning is partitioning the input
domain of a system into finite number of equivalent
classes in such a way that testing one representative
from a class is equivalent to testing any other value
from that class.
To put this in simpler words, since it is practically
infeasible to do exhaustive testing, the next best
alternative is to check whether the program extends
similar behavior or treatment to a certain group of
inputs.
If such a group of values can be found in the input
domain treat them together as one equivalent class
and test one representative from this.
example
Consider a program which takes Salary as
input with values 12000...37000 in the valid
range.
The program calculates tax as follows:
- Salary up to Rs. 15000 No Tax
- Salary between 15001 and 25000 Tax is 18
% of Salary
- Salary above 25000 Tax is 20% of Salary


Accordingly, the valid input domain can be
divided into three valid equivalent classes as
below:
c1 : values in the range 12000...15000
c2: values in the range 15001...25000
c3: values > 25000
However, it is not sufficient that we test only valid test
cases.
We need to test the program with invalid data also as
the users of the program may give invalid inputs,
intentionally or unintentionally.
It is easy to identify an invalid class
c4: values < 12000.
If we assume some maximum limit (MAX) for the
variable Salary, we can modify the class c3 above to
values in the range 25001...MAX
and identify an invalid class c5: values > MAX.
Depending on the system.
Since the input has to be salary it can be seen
intuitively that numeric and non-numeric values are
treated differently by the program. Hence we can
form two classes
- class of non-numeric values
- class of numeric values
Since all non-numeric values are treated as
invalid by the program class c1 need not be
further subdivided.
Class of numeric values needs further
subdivision as all elements of the class are not
treated alike by the program.
Again, within this class, if we look for groups
of values meeting with similar treatment from
the program the following classes can be
identified:
- values < 12000
- values in the range 1200015000
- values in the range 1500125000
- values in the range 25001...MAX
- values > MAX

Equivalent classes identified
summary
To design test cases using equivalence
partitioning, for a range of valid input values
identify
- one valid value within the range
- one invalid value below the range and
- one invalid value above the range
Example
Similarly, To design test cases for a specific set
of values
- one valid case for each value belonging to the
set
- one invalid value
Test Cases for Types of Account (Savings,
Current) will be
- Savings, Current (valid cases)
- Overdraft (invalid case)
It may be noted that we need fewer test cases if
some test cases can cover more than one
equivalent class.

II. Boundary Value Analysis
Even though the definition of equivalence partitioning
states that testing one value from a class is
equivalent to testing any other value from that class,
we need to look at the boundaries of equivalent
classes more closely. This is so since boundaries are
more error prone.
To design test cases using boundary value analysis,
for a range of values,
- Two valid cases at both the ends
- Two invalid cases just beyond the range limits
The test cases using boundary value
analysis are
III. Cause Effect Analysis
The main drawback of the previous two
techniques is that they do not explore the
combination of input conditions.
Cause effect analysis is an approach for
studying the specifications carefully and
identifying the combinations of input
conditions (causes) and their effect in the
form of a table and designing test cases
It is suitable for applications in which
combinations of input conditions are few and
readily visible.
IV. Cause Effect Graphing
This is a rigorous approach, recommended
for complex systems only.
In such systems the number of inputs and
number of equivalent classes for each input
could be many and hence the number of
input combinations usually is astronomical.
Hence we need a systematic approach to
select a subset of these input conditions.
Guidelines for graphing
Divide specifications into workable pieces as it may be
practically difficult to work on large specifications.
Identify the causes and their effects. A cause is an input
condition or an equivalence class of input conditions. An effect
is an output condition or a system transformation.
Link causes and effects in a Boolean graph which is the cause-
effect graph.
Make decision tables based on the graph. This is done by
having one row each for a node in the graph. The number of
columns will depend on the number of different combinations of
input conditions which can be made.
Convert the columns in the decision table into test cases.
Consider the following specification:
A program accepts Transaction Code - 3
characters as input.
For a valid input the following must be true.
1st character (denoting issue or receipt)
+ for issue
- for receipt
2nd character - a digit
3rd character - a digit
Control flow graph
In the graph: (1) or (2)
must be true (V in the
graph to be interpreted
as OR)
(3) and (4) must be
true (? in the graph to
be interpreted as AND)

The Boolean graph has to be
interpreted as follows
- node (1) turns true if the 1st character is +
- node (2) turns true if the 1st character is -
(both node (1) and node (2) cannot be true
simultaneously)
- node(3) becomes true if the 2nd character
is a digit
- node(4) becomes true if the 3rd character is
a digit
- the intermediate node (5) turns true if (1) or
(2) is true (i.e., if the 1st character is + or -
)
- the intermediate node (6) turns true if (3)
and (4) are true (i.e., if the 2nd and 3rd
characters are digits)
- The final node (7) turns true if (5) and (6)
are true. (i.e., if the 1st character is + or -,
2nd and 3rd characters are digits)
- The final node will be true for any valid input
and false for any invalid input.
Node
Some possible combination of node states
1 0 1 1 1 0 1
2 0 0 0 0 0 0
3 0 0 0 1 1 1
4 0 0 1 0 1 1
5 0 1 1 0 0 1
6 0 0 0 1 1 1
7 0 0 0 0 0 1
Sample Test Case for the
Column
$xy +ab -3 +2y @45 67
Node
Some possible combination of node states
1 0 1 1 1 0 1
2 0 0 0 0 0 0
3 0 0 0 1 1 1
4 0 0 1 0 1 1
5 0 1 1 0 0 1
6 0 0 0 1 1 1
7 0 0 0 0 0 1
Sample Test Case for the
Column
$xy +ab -3 +2y @45 67
Node
Some possible combination of node states
1 0 1 1 1 0 1
2 0 0 0 0 0 0
3 0 0 0 1 1 1
4 0 0 1 0 1 1
5 0 1 1 0 0 1
6 0 0 0 1 1 1
7 0 0 0 0 0 1
Sample Test Case for the
Column
$xy +ab -3 +2y @45 67
V. Error Guessing
Error guessing is a supplementary technique
where test case design is based on the
tester's intuition and experience.
There is no formal procedure. However, a
checklist of common errors could be helpful
here.
3.2 White Box Approach
Basis Path Testing
Basis Path Testing is white box testing
method where we design test cases to cover
every statement, every branch and every
predicate (condition) in the code which has
been written.
Thus the method attempts statement
coverage, decision coverage and condition
coverage.
To perform Basis Path Testing
Derive a logical complexity measure of procedural design
Break the module into blocks delimited by statements
that affect the control flow (eg.: statement like return,
exit, jump etc. and conditions)
Mark out these as nodes in a control flow graph
Draw connectors (arcs) with arrow heads to mark the
flow of logic
Identify the number of regions (Cyclomatic Number)
which is equivalent to the McCabes number
Define a basis set of execution paths
Determine independent paths
Derive test case to exercise (cover) the basis set


McCabes Number
(Cyclomatic Complexity)

Gives a quantitative measure of the logical
complexity of the module
Defines the number of independent paths
Provides an upper bound to the number of tests that
must be conducted to ensure that all the statements
are executed at least once.
Complexity of a flow graph g, v(g), is computed in
one of three ways:
V(G) = No. of regions of G
V(G) = E - N +2 (E: No. of edges & N: No. of nodes)
V(G) = P + 1 (P: No. of predicate nodes in G or No. of
conditions in the code)

McCabes Number = No. of Regions (Count the mutually
exclusive closed regions and also the whole outer space as one
region)
= 2 (in the above graph)
Two other formulae as given below also define the above
measure:

McCabes Number = E - N +2
( = 6 6 +2 = 2 for the above graph)

McCabes Number = P + 1
( =1 + 1= 2 for the above graph)
Please note that if the number of conditions is more than one in
a single control structure, each condition needs to be separately
marked as a node.
When the McCabes number is 2, it indicates that
there two linearly independent paths in the code. i.e.,
two different ways in which the graph can be
traversed from the 1st node to the last node. The
independents in the above graph are:
i) 1-2-3-5-6
ii) 1-2-4-5-6
The last step is to write test cases corresponding to
the listed paths.
This would mean giving the input conditions in such a
way that the above paths are traced by the control of
execution.
The test cases for the paths listed here are show in
the following table.
Path
Input
Condition
Expected
Result
Actual
Result Remarks
i)
value of a >
value of b
Increment a
by 1
ii)
value of a
<= value of
b
Increment b
by 1
4. When to stop testing
When To Stop Testing ?
The question arises as testing is never complete and
we cannot scientifically prove that a software system
does not contain any more errors.
Common Criteria Practiced
Stop When Scheduled Time For Testing Expires
Stop When All The Test Cases Execute Without
Detecting Errors
Both are meaningless and counterproductive as the
first can be satisfied by doing absolutely nothing
Stop When

All test cases, derived from equivalent partitioning,
cause-effect analysis & boundary-value analysis, are
executed without detecting errors.
Drawbacks
Rather than defining a goal & allowing the tester to
select the most appropriate way of achieving it, it
does the opposite !!!
Defined methodologies are not suitable for all
occasions !!!
No way to guarantee that the particular methodology
is properly & rigorously used
Depends on the abilities of the tester & not
quantification attempted !
Completion Criterion Based On The Detection Of
Pre-Defined Number Of Errors
In this method the goal of testing is positively
defined as to find errors and hence this is a
more goal oriented approach.
Eg.
- Testing of module is not complete until 3
errors are discovered
- For a system test : Detection of 70 errors or an
elapsed time of 3 months, whichever come
later
How To Determine "Number Of
Predefined Errors " ?

Predictive Models
Based on the history of usage / initial testing & the
errors found
Defect Seeding Models
Based on the initial testing & the ratio of detected
seeded errors to detected unseeded errors
(Very critically depends on the quality of 'seeding')
Using this approach, as an example, we can say that
testing is complete if 80% of the pre-defined number of
errors are detected or the scheduled four months of
testing is over, whichever comes later.
Caution !
The Above Condition May Never Be Achieved
For The Following Reasons
Over Estimation Of Predefined Errors
(The Software Is Too Good !!)
Inadequate Test Cases
Hence a best completion criterion may be a
combination of all the methods discussed
Module Test
Defining test case design methodologies (such
as boundary value analysis...)
Function & System Test
Based on finding the pre-defined number of
defects
5. Debugging
Debugging occurs as a consequence of
successful testing.
It is an exercise to connect the external
manifestation of the error and the internal
cause of the error
Debugging techniques include use of:
Breakpoints
A point in a computer program at which
execution can be suspended to permit manual
or automated monitoring of program
performance or results
Desk Checking
A technique in which code listings, test results
or other documentation are visually examined,
usually by the person who generated them, to
identify errors, violations of development
standards or other problems
Dumps
1. A display of some aspect of a computer
programs execution state, usually the
contents of internal storage or registers
2. A display of the contents of a file or device
Single-Step Operation
In this debugging technique a single computer
instruction, is executed in response to an
external signal.
Traces
A record of the execution of computer
program, showing the sequence of instructions
executed, the names and values of variables
or both.