Professional Documents
Culture Documents
Management Science
Publication details, including instructions for authors and subscription information:
http://pubsonline.informs.org
This article may be used only for the purposes of research, teaching, and/or private study. Commercial use
or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher
approval, unless otherwise noted. For more information, contact permissions@informs.org.
The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness
for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or
inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or
support of claims made of that product, publication, or service.
© 2000 INFORMS
INFORMS is the largest professional society in the world for professionals in the fields of operations research, management
science, and analytics.
For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org
Effects of Process Maturity on Quality,
Cycle Time, and Effort in Software Product
Development
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
product design has begun, but they expect the soft- 1986, Nandakumar et al. 1993). This view is also
ware to be delivered without delay. As a consequence, espoused by advocates for software process improve-
to satisfy customer needs under severe schedule pres- ment (Humphrey 1995). These advocates have pro-
sure and yet control development costs, IT firms may vided frameworks for software firms to characterize
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
deviate from disciplined practices in their develop- the capability of their software development practices
ment process and cut corners by shipping the prod- (El Emam and Goldenson 1996, Paulk et al. 1995). One
ucts without adequate testing (Kemerer 1997). Such of the widely adopted frameworks is the Capability
behavior can result in a higher number of defects at Maturity Model™ (CMM™) developed by the Soft-
the customer site that delay the acceptance of the ware Engineering Institute (SEI) at Carnegie Mellon
product, and the cost of fixing these defects can be University (Paulk et al. 1995). Based on the specific
significant. These dynamics suggest the importance of software practices adopted, the CMM classifies the
delivering software products on time without com- software process into five maturity levels. Analogous
promising quality and cost. to concepts of time-based competition in manufactur-
In manufacturing, reduction of product develop- ing, the basic premise of this framework is that im-
ment cycle time has become the focal point of compe- provements in cycle time, cost, and quality can be
tition in many industries. The fast pace of technolog- simultaneously attained by improving software pro-
ical change and the intensity of international cess capability. These improvements are thought to
competition place a premium on a firm’s ability to arise from reduced defects and rework in a mature
respond quickly to customer demands. Many firms software development process. However, it is impor-
have adopted time-based competition strategies to tant to provide empirical evidence to substantiate
reduce product development time and deliver higher these beliefs.
quality products and services to their respective cus- In this study, we empirically investigate the rela-
tomers at lower cost. Under time-based competition, tionships between process maturity measured on the
firms strive to streamline and constantly improve the CMM maturity scale, product quality, development
reliability and capability of their manufacturing pro- cycle time, and effort for 30 software products created
cesses. A key premise underlying process improve- by a major IT firm over a period of 12 years. Size and
ment in manufacturing is the elimination of waste and design complexity are reported as additional signifi-
rework in manufacturing activities by reducing prod- cant variables that explain quality in the delivered
uct defects (Bockerstette and Shell 1993). software products.
In software development, an important issue is We find that improvements in process maturity lead
whether process improvement pays off in terms of to higher quality but also increased effort in our
higher quality, reduced cycle time, and lower cost. sample of software products. However, higher quality
Under the conventional paradigm, higher quality can in turn leads to reduced cycle time and development
be achieved only at the expense of increased develop- effort in the software products. The net effect of
ment expenditures and longer cycle times. From this improvement in process maturity on development
perspective, effort expended to improve software de- cycle time and effort is negative. At the average values
velopment processes varies with the level of quality for process maturity and software quality, a 1% im-
attained. A typical view of software managers operat- provement in process maturity leads to a 0.32% net
ing from this paradigm is: “I’d rather have it wrong reduction in cycle time, and a 0.17% net reduction in
than have it late. We can always fix it later.” (Paulk et development effort (taking into account the positive
al. 1995, p. 4) direct effects and the negative indirect effects through
An alternative view from manufacturing is that quality). These findings provide empirical support in
quality, cost, and cycle time are complementary, i.e., the context of software production for theories of the
improvements in quality directly relate to improved cycle time and cost benefits of improved quality
cycle time and productivity (Crosby 1979, Deming deriving from process improvement.
The rest of our paper is organized as follows. A brief resources in the initial stages of product design drive
review of the relevant literature is presented in §2. increased quality. However, this study does not con-
Section 3 contains the development of our model and sider the cycle time impact of quality improvement.
theory. Section 4 provides details about the research New technologies are emerging at an increasing rate
site and data collection. Model estimation and results
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
methods of products and process. The CMM practices Cycle-Time ⫽ Function (Process-Maturity,
aid in reducing defect injection and in early identifi-
Product-Quality, Product-Size,
cation of defects. As a consequence, the number of
errors in system and acceptance testing will be lower Requirements-Ambiguity). (2)
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
for software products developed with a mature pro- Cycle-time is the time to develop the product, i.e., the
cess. This implies: elapsed time in days from the start of design on the
Hypothesis 1 (Process maturity and product product until its final acceptance by the customer. The
relationship between cycle time, process maturity, and
quality). Higher levels of process-maturity lead to higher
quality has been viewed from two different perspec-
product-quality in software products.
tives. One view is that cycle time must be traded off
We control in this model for the effect of product against improvements in quality. That is, increased
size (product-size) and product design complexity investment in process improvement requires addi-
(product-design-complexity). Product-size (defined in tional time for testing, code inspections, etc., that add
terms of thousand lines of source code in the product) to the elapsed time required to develop the product
is a factor that we expect to be inversely related to and delay its introduction.
product quality. Prior studies of software productivity In contrast, an alternate view is that improvements
and quality have identified product size measured in in quality can lead to an overall reduction in cycle
lines of code as a primary determinant of product time. The rationale behind this is that by adopting
defects (Basili and Musa 1991). Because of the volume disciplined practices as specified in the CMM frame-
of code and increased product functionality, larger work and improving process maturity, the time spent
products provide more opportunity to introduce er- on preventing and uncovering defects early in the
rors. In addition, larger products may include more development can be more than recovered by avoiding
modules and interactions between modules, thus fur- rework to correct defects detected at the later stages of
ther increasing the possibility of defects. Product- product development (Jones 1997). In the context of
design-complexity in our model captures three impor- software production, several studies have indicated
tant dimensions of complexity: domain, data, and that more time is needed to fix defects uncovered at
decision complexity. Domain complexity reflects the later stages in software development than in earlier
difficulty of the functionality, specifically the algo- stages (e.g., Swanson et al. 1991, Abdel-Hamid and
rithms and calculations that will be accomplished in a Madnick 1991). Following this view, we hypothesize
product. Data complexity refers to the complexity of that increased investment in process improvement
the data structures and relationships in the software activities (as reflected in process maturity) can in-
product. Decision complexity refers to the complexity crease cycle time. However, higher process maturity
of decision paths and structures in the software prod- leads to higher product quality, and as products
uct. The more complex the product design on these exhibit fewer defects, there is less rework, thereby
dimensions, the higher the likelihood that errors will reducing cycle time. Therefore:
be injected in development and uncovered in system Hypothesis 2 (Process maturity and cycle time).
and acceptance testing when live data rather than test Higher levels of process-maturity are associated with in-
cases are used (Munson 1996). creased cycle-time in software products.
Hypothesis 3 (Product quality and cycle time).
3.2. Cycle Time
In our second model (2), we examine the link Higher product-quality is associated with lower cycle-time
in software products.
between process maturity, quality, and develop-
ment cycle time in software products, controlling In the cycle time model, we control for the effects of
for the effects of product size and the ambiguity of product size (product-size) and the ambiguity of user
user requirements. requirements (requirements-ambiguity). We expect that
product size will be positively related to cycle time, Along these lines, we expect that improved quality is
ceteris paribus, because larger products involve more associated with lower development effort. However,
work and should take longer to develop in the absence higher levels of process maturity would directly in-
of variations in schedule pressure. Products for which crease development effort as a result of the additional
user requirements are not clearly specified in design activities for process improvement. Thus:
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
Requirements-Ambiguity). (3)
4. Research Design and
Effort refers to the person-months required to de-
velop the software product. The relationship between
Methodology
development effort, process maturity, and product 4.1. Research Setting
quality has also been discussed from two different We examine data collected on 30 software products
viewpoints. The conventional school of thought as- created by the systems integration division of a $1
serts that effort and quality improvement must be billion per year IT firm. The products comprise ap-
counterbalanced. That is, increased effort is required proximately 3.3 million lines of COBOL code and were
to follow disciplined practices, use quality tools, and created from 1984 to 1996. The products were devel-
conduct rigorous testing and code reviews in attaining oped as part of a $230 million effort to build a material
higher quality levels. A different school of thought requirements planning (MRP) information system to
views development effort and quality improvement as manage spare parts acquisition. All of the MRP soft-
complementary. This stems from the argument that ware is designed to satisfy a stringent five-second
defects detected at the later stages of product devel- response time for all queries, regardless of geographic
opment lead to substantial rework, and the cost of this location or complexity of retrieval.
rework could be significantly higher than investments This IT firm provides an interesting research site to
in quality improvement at the early stages of product study software development because the firm focused
development (Boehm 1981). This implies that there is on improvements to development processes and qual-
an inverse relationship between quality and effort. ity during the time frame in which the products were
developed, and advanced in process maturity. A con- All other data were collected by the IT development
tractual arrangement using incentive targets moti- firm and were audited by the customer to ensure
vated the IT firm to continually search for techniques accuracy and accountability. Within the IT firm, the
to improve the process and reduce cycle time and departments responsible for data collection included
costs. Costs exceeding a fixed ceiling were borne by
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
manage the development process. Domain, data, and acceptance test. Development effort is the total number
decision complexity are three variables that were of person-months logged by the development team in
critical to the estimation process and were principal all the stages of product development, starting from
inputs to the Software Productivity Quality and Reli- initial design through final product acceptance testing.
ability (SPQR20娀) methodology for projecting sched-
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
on a scale of 1 to 5, with 5 reflecting the highest degree addition, for the data sample in this study, statistical
of ambiguity in user requirements specification, and 1 tests rejected standard assumptions of the linear model
indicating that user requirements are clear and unam- such as normality of error terms. Thus, a generic multi-
biguous. Descriptive statistics for all variables are in plicative specification for our models is adopted through
Table 1, and a correlation matrix is in Table 2. log transformation of the variables (Kmenta 1986). The
specification test for nonnested models (the J-test) sup-
5. Analysis and Results ported the log-linear models specified below over linear
A linear specification of the three models implies that the models for each of our equations (Davidson and Mac-
effects of process maturity, product size, product design Kinnon 1995).
complexity, and ambiguity of user requirements on
ln共Product-Quality兲 ⫽  01 ⫹  11
product quality, cycle time, and development effort are
additively separable and linear. However, in the case of ⴱ ln共Process-Maturity兲
software development, the effects of size and other
factors are not linear. Both economies and diseconomies ⫹  21 ⴱ ln共Product-Size兲 ⫹  31
of scale (i.e., size) of the software product have been ⴱ ln共Product-Design-Complexity兲 ⫹ ⑀ 01 (1)
observed by researchers for cost (Banker and Slaughter
1997, Banker et al. 1994) and quality (Newfelder 1993). In ln共Cycle-Time兲 ⫽  02 ⫹  12 ⴱ ln共Product-Quality兲
As a robustness check, we estimated a rank regres- acceptable limit. We did not detect any influential outli-
sion model for each equation (Iman and Conover ers in the equations using Cook’s distance (Cook and
1979). The parameter estimates from the rank regres- Weisberg 1982) and the guidelines specified by Belsley et
sions (column six in Tables 3 through 5) are similar in al. (1980). Because the products were developed over a
sign, magnitude, and significance to those from the period of 12 years, we tested for serial correlation be-
OLS regressions, suggesting the robustness of the OLS tween products using both the Durbin-Watson test
estimates. (Durbin and Watson 1971) and the Breusch-Godfrey test
Thus, we used the OLS estimates for the interpretation (Breusch and Godfrey 1981) but did not find any evi-
of our results. Standard assumptions of the OLS estima- dence of it. The calculated values of all four models’
tors were tested. The assumption of normality is not F-statistics exceeded the critical values at the 5% signif-
rejected for any of the models at the 5% significance level icance level.
using the Shapiro-Wilk test (Shapiro and Wilk 1965). The
presence of heteroskedasticity in all the models was
tested using White’s (1980) test, and no evidence of it
6. Discussion
was found. The effect of multicollinearity was examined 6.1. Product-Quality Model
using conditions specified in Belsley et al. (1980). The In model one, we find as expected that higher levels of
condition index for all models is less than 30, within the process maturity are associated with higher product
quality (i.e., fewer defects in system and acceptance higher quality. Although larger products may have
testing,  11 ⫽ 1.589, p ⬍ 0.001). The value of the more defects, in our sample the rate of increase in
coefficient for process maturity implies that, holding defects resulting from size is lower for larger products.
the other variables in the equation constant at their Hence the coefficient of size in our model is positive
mean levels, a 1% improvement in process maturity is (  21 ⫽ 0.234, p ⫽ 0.020). Also as we expected,
associated with a 1.589% increase in product quality. products with a more complex design have lower
The value of the process maturity coefficient also quality.
suggests that the quality benefits increase with higher
levels of process maturity. As noted earlier, if the 6.2. Cycle-Time Model
development process is not disciplined and mature, Our findings in this model indicate that product
there will be deviations from the standard process quality is significantly and negatively associated with
such as absence of requirements tracking, lack of cycle time (  12 ⫽ ⫺0.454, p ⫽ 0.004). That is, higher
rigorous code reviews and inspections, and myopic quality products exhibit significant reductions in cycle
design choices. As a consequence, both design and time. The value of the coefficient for product quality
coding errors will be injected during the development implies that, holding the other variables in the equa-
process. tion constant at their mean levels, a 1% improvement
Our results indicate that larger products exhibit in product quality is associated with a 0.454% decrease
in cycle time. Because the model specification is mul- quent reduction in effort because of improved quality,
tiplicative, the value for the coefficient of product the net effect is negative (⫺0.175). As expected, we
quality implies that, at higher levels of quality, the find that larger products are associated with greater
payoff in development cycle time savings decreases. effort. The coefficient for requirements ambiguity is
Our results also indicate that controlling for product not statistically significant in this model.
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
Figure 4 Marginal Effects of Product Quality on Cycle Time Table 6 Estimated Benefits of Process Maturity
275–282.
software engineering research and practice. A primary , S. Slaughter. 1997. A field study of scale economies in software
contribution is the rigorous examination of the inter- maintenance. Management Sci. 43(12) 1709 –1725.
relationships between quality, cycle time, and effort in Basili, V. R., J. D. Musa. 1991. The future engineering of software: A
software product development. As noted earlier, prior management perspective. IEEE Comput. 20(4) 90 –96.
, B. Perricone. 1984. Software errors and complexity: An empir-
studies have modeled quality, cycle time, or effort in
ical investigation. Comm. ACM 27(1) 42–52.
the absence of one or more of the other variables. Belsley, D. A., E. Kuh, R. E. Welsch. 1980. Regression Diagnostics:
Another contribution is the examination of the direct Identifying Influential Data and Sources of Collinearity. Wiley and
and indirect effects of improved process maturity on Sons, New York.
quality, cycle time, and development effort. These Bockerstette, J., R. Shell. 1993. Time Based Manufacturing. McGraw-
effects are not well understood and are generally not Hill, New York.
Boehm, B. W. 1981. Software Engineering Economics. Prentice-Hall,
rigorously studied in the software engineering litera-
Englewood Cliffs, NJ.
ture. Further research could explore reengineering the Bohn, R. E. 1995. Noise and learning in semiconductor manufactur-
software life cycle through leaner processes, enabling ing. Management Sci. 41(1) 31– 42.
greater reduction in cycle time from quality improve- Breusch, T., L. Godfrey. 1981. A review of recent work on testing for
ment. Also, future research could examine the trade- autocorrelation in dynamic simultaneous models. D. Currie, R.
off between cycle time and effort, controlling for team Nobay, D. Peel, eds. Macroeconomic Analysis: Essays in Macro-
economics and Econometrics. Croon Helm, London, England.
loading and schedule pressure. It would be instructive
63–105.
to explore the effects of process improvement in Brooks, F. 1995. The Mythical Man-Month: Essays on Software Engi-
different phases of the software life cycle. Finally, the neering. Anniversary Edition, Addison-Wesley, Reading, MA.
impact of process improvement on the nonengineer- Conte, S. D., H. E. Dunsmore, V. Y. Shen. 1986. Software Engineering
ing activities that support software development Metrics and Models. Benjamin/Cummings Publication Com-
could be examined. pany, Menlo Park, CA.
Cook, R. D., S. Weisberg. 1982. Residuals and Influence in Regression.
For software engineering practice, our results quan-
Chapman & Hall, London, England.
tify the benefits of process maturity. Software manag-
Crosby, P. B. 1979. Quality Is Free. McGraw-Hill, New York.
ers may be reluctant to invest in quality improvement Davidson, R., J. MacKinnon. 1995. Several tests for model specifica-
practices without knowledge of the return on that tions in the presence of multiple alternatives. Econometrica 49
investment. For managers of software projects, our 781–793.
analysis provides a methodology for quantifying the , . 1993. Estimation and Inference in Econometrics. Oxford
University Press, New York.
time and cost savings from quality improvement.
Deming, W. E. 1986. Out of the Crisis. MIT Center for Advanced
These results provide useful insights for planning and
Engineering Study, Cambridge, MA.
assessing the return on investment from quality im- Durbin, J., G. Watson. 1971. Testing for serial correlation in least
provement in the development process. 1 squares regression III. Biometrica 58 1– 42.
El Emam, K., D. R. Goldenson. 1996. Some initial results from the
1
The authors gratefully acknowledge research support from the international SPICE trials. Software Process Newsletter, Technical
Graduate School of Industrial Administration at Carnegie Mellon Council on Software Engineering 6 (Spring) IEEE Computer
University and the University of Michigan Business School. The Society.
assistance of managers and staff at our data site were invaluable. Fenton, N. E., S. L. Pfleeger. 1997. Software Metrics: A Rigorous and
Helpful comments were provided by participants in research sem- Practical Approach. International Thompson Computer Press,
inars at Carnegie Mellon University, the University of Michigan, the London, England.
University of Pennsylvania, the University of Minnesota, the Uni- Gopal, A., T. Mukhopadhyay, M. S. Krishnan. The role of software
versity of Texas at Austin, and the Workshop on Information process and communication in offshore software development.
Systems and Economics (WISE ’97). Comm. ACM. Forthcoming.
Greene, W. H. 1997. Econometric Analysis. 3rd ed., MacMillan Lyu, M. R. 1996. Handbook of Software Reliability Engineering.
Publishing Company, New York. McGraw-Hill, New York.
Hausman, J. 1978. Specification tests in econometrics. Econometrica Mowrey, D. C. 1996. The International Computer Software Industry: A
46 1251–1271. Comparative Study of Industrial Evolution and Structure. Oxford
Herbsleb, J., D. Zubrow, D. Goldenson, W. Hayes, M. Paulk. 1997. University Press, Oxford, U.K.
Downloaded from informs.org by [193.0.65.67] on 18 December 2014, at 07:00 . For personal use only, all rights reserved.
Software quality and the capability maturity model. Comm. Munson, J. 1996. Software faults, software failures, and software
ACM 40(6) 30 – 40. reliability modeling. Inform. Software Tech. 38(11) 687– 699.
Humphrey, W. S. 1995. A Discipline for Software Engineering. Addi- Nandakumar, P., S. M. Datar, R. Akella. 1993. Models for measuring
son-Wesley, Reading, MA. and accounting for cost of conformance quality. Management
Iman, R., W. Conover. 1979. The use of the rank transform in Sci. 39(1) 1–16.
regression. Technometrics 21(4) 499 –509. Newfelder, A. M. 1993. Ensuring Software Reliability. Marcel Dekker,
Jones, C. 1986. How not to measure programmer productivity. Inc., New York.
Computerworld 20(2) 65–76. Paulk, M. C., C. V. Weber, B. Curtis, M. B. Chrissis. 1995. The
1996. Applied Software Measurement: Assuring Productivity and Capability Maturity Model: Guidelines for Improving the Software
Quality. McGraw-Hill, New York. Process. Addison-Wesley Publishing Company, Reading, MA.
1997. Software Quality: Analysis and Guidelines for Success. ITP Pfleeger, S. 1998. Software Engineering: Theory and Practice. Prentice-
Press, London, UK. Hall, NJ.
Kaplan, R. 1986. Must CIM be justified by faith alone? Harvard Bus. Shapiro, S., M. Wilk. 1965. An analysis of variance test for normality.
Rev. 64(2) 87–95. Biometrika 52 591– 612.
Kemerer, C. F. 1997. Software Project Management Readings and Cases. Swanson, K., D. McComb, J. Smith, D. McCubbrey. 1991. The
McGraw-Hill, New York. application software factory: Applying total quality techniques
Kmenta, J. 1986. Elements of Econometrics. Macmillan, New York. to systems development. MIS Quart. 15(4) 566 –580.
Krishnan, M. S. 1996. Cost and quality considerations in software Tagaras, G., H. Lee. 1996. Economic models of vendor evalua-
product management. Ph.D. Dissertation, Graduate School of tion with quality cost analysis. Management Sci. 42(11) 1531–
Industrial Administration, Carnegie Mellon University, Pitts- 1543.
burgh, PA. The Wall Street Journal. 1997. The spoils of war. September 11.
, S. Kekre, C. H. Kriebel, T. Mukhopadhyay. 2000. An empirical White, H. 1980. Heteroskedasticity-consistent covariance matrix
analysis of productivity and quality in software products. estimator and a direct test for heteroskedasticity. Econometrica
Management Sci. Forthcoming. 48(5) 817– 838.