Taguchi-Based Metrics for Software Quality

Kiumi Akingbehin, PhD Department of Computer and Information Science University of Michigan-Dearborn Dearborn, Michigan 48128, USA Email: kiumi@umich.edu Abstract
A set of metrics (numerical software measurements) based on the Taguchi philosophy are proposed for measuring software quality. The Taguchi approach, originally developed for manufacturing processes, define quality in terms of loss imparted to society by a product after delivery of the product to the end user. The nominal-the-best and smaller-the-best loss functions developed by Taguchi are adapted for software products and case studies are presented to support the adaptation. The proposed Taguchi-based metrics are in line with six-sigma statistical quality control techniques that have been used with software products in recent years. Experiences with the proposed metrics have shown good correlation with existing popularly known software quality metrics. Whilst the Taguchi approach has many advantages, the primary advantage is the ability to quantify quality in monetary values, such as dollars in the USA or Euros in Europe. The Taguchi philosophy is heavily used in the automotive industry and there is general consensus that it has played a major role in automotive quality improvements in the last decade. This paper applies the Taguchi philosophy to software products. In particular, two loss functions, as defined by Taguchi, are discussed in a software context and case studies are presented.

2. Software Product Quality Metrics

Software metrics can be classified into one or more of the following three categories [2] [8]: product metrics process metrics project metrics Product Metrics describe product characteristics such as: size, complexity, performance, and quality. Process metrics apply primarily to software development and maintenance activities and include: development time, defect removal effectiveness, and time to fix an error. Project metrics describe project characteristics and execution, and are of particular interest to the project manager. They include human resource, cost, time, schedule, and productivity Some example software quality metrics are: availability (AVAIL) reliability (REL) defect rate (e.g. DEFECT/KLOC) problems per user month (PUM) mean time between failure (MTBF) The MTBF is considered by some to be a misnomer. An alternative term that has been proposed is MTBDF, for mean time between detection of failure. Consistent with most software quality metrics,

1. Introduction
Software metrics continue to play a major role in all areas of software engineering. Corporate advancement to higher CMMI levels mandates the use of metrics. The need to develop quality software products within cost and time limitations continues to be a high priority and it too invariably mandates the use of metrics for resource and time estimations. Much progress has been made in this respect, but losses are still enormous. In the U. S. alone, recent statistics indicate approximately $59 billion in cost overruns on software projects annually, and an additional $81 billion on canceled software projects annually [9]. The Taguchi philosophy [3] [21] measures the quality of a product as the loss imparted to society by the product after delivery of the product to the user. While developed originally in a manufacturing environment, the philosophy does not preclude software products. It is therefore surprising that there has been limited application of the Taguchi philosophy to software products. This is more so when one considers the fact that the related statistical concept of six-sigma analysis has been extensively applied to software products in recent years [13].

Proceedings of the Fourth Annual ACIS International Conference on Computer and Information Science (ICIS05) 0-7695-2296-3/05 $20.00 2005 IEEE

Taguchi-based metrics are primarily product quality metrics.

If $A is the cost of repairing or discarding the product when y is at a limit, then the quality loss coefficient is given by: k = A / ( y - t ) 2 = A / dev 2 The loss function can also sample from a population: be determined using a

3. Loss to Society
There are many ways in which a software product can contribute to Taguchis philosophy of loss imparted to society. Consider the following five examples: 1. A software product that is so unsatisfactory that it has to be replaced with a new product. The loss to society is the cost of replacing the software product; primarily the cost of the new product. 2. A software product that is so difficult to use that it results in loss of time and consequently loss of productivity. The loss to society is the labor cost of the lost time. 3. A software product that has such a long learning curve that considerable time is lost in learning to use the product. The loss to society is the labor cost for the extra learning time. 4. A software product that fails frequently and requires fixing at each failure. The loss to society is the cumulative cost of fixing the software. 5. A software product that crashes frequently but does not require fixing at each failure. The loss to society is the labor cost that is equivalent to the cumulative time to recover from each crash and continue with the work that was interrupted.

Lave = k [ z 2 + ( ymean - t ) 2 ] where: Lave = average (or expected) loss k = quality loss coefficient z = sigma, population standard deviation, not usually known s = sample standard deviation, usually used instead of z ymean = performance characteristic t = target value Note that for n samples, mean and standard deviation are given by: mean, xmean = (Sum xi ) / n std dev, z = [ (Sum ( xi xmean) 2 ) / ( n 1) ] 1/2

5. Smaller-the-Best Loss Function

The smaller-the-best loss function is for a property whose value should be as small as possible. It can be obtained from the nominal-the-best loss function by substituting zero for t, the target value, as follows: L = k ( y - t) 2 = ky2 where: L = lost incurred as quality parameter deviates from target y = performance characteristic (value of interest) t = target value (ideal value) k = quality loss coefficient

4. Nominal-the-Best Loss Function

To estimate the loss imparted to society by a product, Taguchi has developed over 68 loss functions [3] [21] for different situations. The most commonly used loss function is the quadratic function commonly known as the nominal-the-best loss function. The nominal-thebest loss function applies to a performance parameter that should be maintained at a certain value and is given by: L = k(y-t)2 where: L = lost incurred as quality parameter deviates from target y = performance characteristic (value of interest) t = target value (ideal value) k = quality loss coefficient

6. Case Study 1: Nominal-the-Best

A software product is designed to maintain the cruising speed of an aircraft at (640, +/-10) mph for maximum fuel economy. For a typical trip, the additional fuel cost incurred at the limit is $1700. (a) determine the loss coefficient (b) give the loss function (c) determine the loss at 646 mph

Proceedings of the Fourth Annual ACIS International Conference on Computer and Information Science (ICIS05) 0-7695-2296-3/05 $20.00 2005 IEEE

Applying the nominal-the-best loss function, we obtain: (a) Loss coefficient, k = A/(yt)2 = 1700 / ( 650 640 ) 2 = 17 (b) Loss function, L = f (y), is given by, L = 17 ( y 640 ) 2 (c) Loss at 646 mph, L = 17 ( y 640 ) 2 = 17 ( 646 640 ) 2 = 17 X 36 = $612.00 A software product is designed to maintain the cruising speed of an aircraft at 640 mph for maximum fuel economy. The loss coefficient for a typical trip is 231, and a sampling of actual cruising speed is: 636, 640, 638, 639, 643, 639, 646, 642 mph. What is the average loss per trip due to this software controller? Applying the nominal-the-best loss function for samples from a population, we obtain: Mean Cruising Speed, ymean = (636+640+638+639+643+639+646+642) / 8 = 5123 / 8 = 640.375 Sample Standard Deviation, s = [ (Sum ( yi ymean) 2 ) / ( n 1) ] 1/2 = [ (19.14+0.14+5.64+1.89+6.89 +1.89+31.64+2.64) / 7 ] 1/2 1/2 = [69.87/7] = 9.98143 1/2 = 3.15934 Average Loss, Lave = k [ z + ( ymean - t) 2 ] = 231 [ 9.98143 + ( 640.375 640 ) 2 ] = 231 * 10.122055 = $2338.1947

Applying the smaller-the-best loss function, we obtain: (a) Loss coefficient, k = L/y2 = $4179 / 0.92 = 5159.2 $/defect2 (b) Loss function, L = f (y), is given by, L = 5159y2 (c) Loss at 1.1 defect/KLOC, L = ky2 = 5159.2 X 1.12 = $6242.7 Alternatively, a sampling of defect rates can be used. A sampling of defect rates for a company is: 0.9, 0.2, 0.3, 0.5 and 0.8 defect/KLOC. The loss coefficient for the company is 3750. What is the average loss (to society) of software produced by the company? Applying the smaller-the-best loss function for samples from a population, we obtain: Mean Defect Rate, ymean = (0.9+0.2+0.3+0.5+0.8) / 5 = 2.7 / 5 = 0.54 Sample Standard Deviation, s = [ Sum ( yi ymean )2 / ( n 1 ) ] 1/2 = [ ( 0.1296+0.1156+0.0576 +0.0016+0.0676 ) / 4 ] 1/2 1/2 = [ 0.372 / 4 ] = 0.093 1/2 = 0.304959 Average Loss, Lave = k [ z + ( ymean - t) 2 ] = k [ z + ymean2 ] = 3750 ( 0.093 + 0.2916 ) = $1442.25

7. Case Study 2: Smaller-the-Best

Software produced by a CMMI level 3 company has an average defect rate of 0.7 defect/KLOC. The overall cost to fix software produced by the company at the limiting defect rate of 0.9 defect/KLOC is $4179. (a) What is the loss coefficient for software produced by this company? (b) What is the loss function? (c) What is the loss at 1.1 defect/KLOC?

8. Conclusion
This paper has applied the Taguchi definition of quality as loss imparted to society to generate new software quality metrics. In particular, the nominalthe-best and smaller-the-best loss functions were applied to specific case studies. The application is in line with six-sigma based approaches to software quality. Experiences with both the nominal-the-best and smaller-the-best loss functions have shown good correlation to existing software quality metrics.

Proceedings of the Fourth Annual ACIS International Conference on Computer and Information Science (ICIS05) 0-7695-2296-3/05 $20.00 2005 IEEE

Considering the many loss functions (over 68) already developed by Taguchi, this approach to quantifying software quality holds much promise for continuing and future applications in improving software quality.

[12] Meredith, J. R. and Mantel, S. J., Project Management: A Managerial Approach, 4th Edition, Prentice Hall, 2000. [13] Pande, P. and Holpp, L., What is Six Sigma?, McGraw-Hill, 2002. [14] Peters, J. F. and Pedrycz, W., Software Engineering: An Engineering Approach:, John Wiley, 2000. [15] Pfleeger, S. L., Software Engineering:Theory and Practice, 2nd Edition, Prentice Hall, 2001. [16] Poore, J. H., Mills, H. D., and Mutchler, D., Planning and Certifying Software System Reliability, IEEE Software, Vol. 10, No.1, January 1993, pp. 88099. [17] Pressman, R. S., Software Engineering: A Practitioners Approach, 6th ed., McGraw-Hill, 2005. [18] Rumbaugh, J. et al., Object-Oriented Modeling and Design, Prentice-Hall, 1991 [19] Schach, S. R., Classical and Object-Oriented Software Engineering: with UML and Java, 4th Edition, McGraw Hill, 1999. [20] Sommerville, I., Software Engineering,7th ed., Addison-Wesley, 2004. [21] Taguchi, G., Introduction to Quality Engineering, Tokyo: Asian Productivity Organization, 1986.

9. References and General Bibliography

[1] Abani, K. and Akingbehin, K., A Modern Software Engineering Course, ASEE NCS Annual Conference, Rochester, April 2002. [2] Akingbehin, K., A Framework for Software Engineering Metrics, ACIS 4th International Annual Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, SNPD03, Lubeck, October 2003. [3] Besterfield, et al, Total Quality Management, Prentice-Hall, 1995. [4] Bruegge, B. and Dutoit, A. H., Object-Oriented Software Engineering: Conquering Complex and Changing Systems, Prentice Hall, 2004. [5] Cockburn, A. and Highsmith, J., Agile Software Development: The People Factor, IEEE Computer, Vol. 34, No. 11, November 2001. [6] Donaldson, S. E. and Siegel, S. G., Successful Software Development, Prentice-Hall, 2001. [7] Hamlet, D. and Maybee, J., The Engineering of Software: Technical Foundations for the Individual, Addison-Wesley, 2001. [8] Kan, Stephen, Metrics and Models in Software Quality Engineering, 2nd Edition, Addison-Wesley, 2003. [9] Klein, G., Jiang, J., and Tesch, D., Wanted: Project Teams with a Blend of IS Professional Orientations, Communications of the ACM, Vol 45, #6, June 2002. [10] Maxim, K., th Contemporary Software Development Trends, 17 Annual International Conference, Computer Science Division, Association of Management, San Diego, August 1999. [11] Maxim, B. R. and Akingbehin, K., Function Point Metrics for Contemporary Software th Methodology, 14 International Conference on Computers and their Applications, CATA-99, Cancun, Mexico, April 1999. B. R. and Akingbehin,

Proceedings of the Fourth Annual ACIS International Conference on Computer and Information Science (ICIS05) 0-7695-2296-3/05 $20.00 2005 IEEE