You are on page 1of 4

september 2000

Conclusion to identify other measurements that will help


In software development there are many real- these processes and your organization mature.
world problems with solutions that are facili-
tated by measuring the right things at the right
About the Author
time in the process. Process-improvement pro-
Arlene Minkiewicz is the chief scientist at
grams need to include measurement not only
Price Systems LLC. In this role, she is respon-
as the means for identifying the areas for
sible for the research and analysis necessary to
improvement but also as the means to deter-
keep the Price Estimating Suite responsive to
mine progress against improvement goals. If
current cost trends. She has more than 15
you are responsible for a measurement program
years of experience with Price, designing and
or process-improvement program, the first thing
implementing cost models. Prior to her cur-
you need to do is determine what problems
rent assignment, Ms. Minkiewicz functioned
need solving the most, and then start measuring.
as the lead of the Product Enhancement team
The problems presented here are by far, not with responsibility for the maintenance and
inclusive of all the challenges you are taking enhancement of all the Price products. She
on with your improvement program. Chances speaks frequently on software measurement
are really good if you are just starting out with and estimating and has published articles in
measurement and process improvement, that Software Development magazine and the
some of these problems are on your list. And British Software Review. Ms. Minkiewicz has
the measurements and metrics presented here a BS in electrical engineering from Lehigh
will not offer completely comprehensive solu- University and an MS in computer science
tions to these problems — they are presented from Drexel University. She is a member of
as good starting points. the International Society of Parametric
Analysts and International Function Point
Once you get comfortable with the role of
Users Group.
measurement in your processes, you will begin

How to Select Software Project


Macro-Estimation Tools
by Stan Rifkin
Over the years, I have devel- Park’s Fig. 5-1 (shown here in Figure 4) in
oped a checklist for selecting Software Cost and Schedule Estimating: A
automated tools to perform Process Improvement Initiative (SEI-94-SR-03,
macro software project estimates. Mine is May 1994) is one of the most-cited examples
similar, though not identical, to Robert Park’s of an overall software macro-estimating
Checklists and Criteria for Evaluating the Cost process.
and Schedule Estimating Capabilities of
Basically, for the estimation process to support
Software Organizations (SEI-95-SR-005,
a commitment process, the following has to
January 1995).
happen:
There is a zeroth item, one before selecting
and evaluating an estimation tool: establish à Commitments have to be based on the
your project commitment process and then the work to be performed; therefore, there
part within that that focuses on estimation. must be agreement on this.
Park, formerly at the Software Engineering
à Estimates have to be based on (a) the work
Institute (SEI), has written a number of SEI
to be performed and (b) historical records
technical and special reports on the subject of
of performance.
estimation processes, and I recommend the
reader to them. The reports are available for à Commitments must not exceed the capabil-
free on SEI’s Web site (www.sei.cmu.edu/ ity to perform or else there is no reason to
publications/) and are cited at the end of this estimate.
article.
© 2000 Cutter Information Corp. For subscriptions call +1 800 964 5118 or +1 781 641 5118 13
vol. VI, no. 9

Figure 4 — Graphic template: parametric estimating. (Source: Software Engineering Institute)

Incidentally, these form the basis of the looking to see if duration = effort/resources.
Capability Maturity Model (CMM), particu- It is built into every project management
larly the lower levels of software process software package (such as Microsoft
maturity. CMM is about meeting commit- Project), and it states that the time it takes to
ments, and one of the key observations is complete a project is the person-days
that it is easier to meet rationally justified required divided by the number of people.
commitments via estimates based on reason So, if I have a 10-person-day project and 2
and actual historical performance than to people, it would take 5 days. If I had 10
spend time creating stories about why we are people, it would take 1 day.
late, over budget, etc.
If this formula is in the tool, I would have to
With a commitment process and estimation reject it, as it is well known that the relation-
subprocess in hand, and the admonition to ship among duration, effort, and resources is
select a tool that is consistent with both, here not linear. It may be true in building con-
is my checklist of criteria, in priority order. struction that doubling the number of carpen-
The items at the top of the list dominate, so ters cuts their duration in half, but doubling
failing those top criteria can disqualify a can- the number of programmers probably stops
didate tool from further consideration. the software project!
Tools such as SLIM, Price S, and all genera-
Checklist for Tool Selection tions of COCOMO publish their formulas, so
The following items are included on at least I am not going to be surprised!
my checklist of issues I look at when
2. It accurately estimates completed
selecting tools.
projects.
1. The underlying algorithm is published
We use a few typical completed projects to
in the public domain.
see if the candidate tool estimates them
I am worried about surprises, so I want to accurately in retrospect. Clearly, if the tool
read about the formulas used, assumptions cannot accurately estimate our type of
made, constraints, etc. I am particularly

14 Join our free weekly e-mail service, The Cutter Edge: www.cutter.com/consortium/
september 2000
work, we do not want to use it on real, been accomplished by any team anywhere.
future projects. Clearly there is a region or range of project
values (duration, effort, rate of adding peo-
The process of estimating completed
ple, quality, and number of features) such
projects, what Park in the diagram calls
that it is impossible to accomplish a project
“inverse model execution,” leads right to
with that combination. (See Measures for
the next criterion.
Excellence, Putnam and Myers, p. 95.) I
3. I don’t want to subjectively “guess” at want to know that set of values so that I am
the values of variables. sure to steer clear. It is an added bonus if the
tool tells me explicitly what the impossible
If the actual project performance does not
region is so that I don’t have to spend time
equal the estimate then, naturally, I want to
guessing, like playing Minesweeper!
know where I went wrong. Some tools use
scales to assess items such as team experi- 6. The tool takes into account the effects of
ence, by selecting a number between, say, schedule compression.
1 and 10. There may be text associated with
We all know the quip about nine women
each value, such as “new team,” “worked
making a baby in a month (this is a variant
together for many years,” etc. I cannot
of duration = effort/resources). In software
measure team experience, I can only guess at
projects we are commonly asked to produce
its value. When actual does not equal esti-
the minimum duration estimate. In other
mate, I do not know which subjective esti-
words, what is the effort required for the
mates are off and by how much, so I eschew
stated requirements such that the project will
them. I only want to work with items that I
finish in the shortest time? Candidate tools
can objectively measure during the conduct
that simply use a formula to compute the
of a project. I would be especially disturbed
duration (such as the old COCOMO) might
by long lists of such subjective questions, as
not take into account the nonlinear effect of
I would have no hope of tying my responses
schedule compression, that reducing the
to actual project performance that I cannot
duration by 10% can increase the effort by
measure.
40%. Trying to reduce schedule by 20% is
4. The assumptions of the tool mirror my even worse. Michael Mah referred to this in
realities. previous issues of ITMS as the 200/20/6x
rule, whereby doubling the effort (200%)
Estimates are application-area specific (see
results in only about a 20% schedule com-
Chapter 1 in Measures for Excellence:
pression, but with a severe reliability penalty
Reliable Software on Time, Within Budget by
— a 6x rise in defects.
Lawrence Putnam and Ware Myers, Prentice
Hall, 1992), so we need to be sure that the One can see, too, that the formula duration =
candidate tool has been particularized for my effort/resources has the effect of not model-
application type (e.g., avionics, process con- ing compression, which is yet one more rea-
trol, business). It also has to be applicable to son to avoid tools that use it. One of the
the size, programming language, lifecycle, reasons I like tools that model compression
and degree of centralization and formaliza- is that we are commonly given the delivery
tion present in my projects. Often these date during the commitment process, so I
aspects are not stated in estimation tools, so need to be able to ask and answer whether I
they are disqualified from consideration. can stand (that is, manage) the compression
One nameless tool uses a lifecycle that I do in the dictated schedule.
not, so I know that the estimates will not
7. I want a range, not a point, estimate
mirror my reality.
and the probability of achieving it.
5. The tool will not generate an impossible
We all know that estimates are most useful if
schedule.
they give us a range and probability, such as
This was a common response to COCOMO “There is a 70% chance of slight rain, and a
and its progeny. Because it was a formula, 50% chance of significant rain.” In order to
it could compute a duration that was impossi- intelligently make software project decisions
ble in empirical terms. That is, it could we need the same thing. We need to know
compute a duration that had never before the probability or risk for each feasible band

© 2000 Cutter Information Corp. For subscriptions call +1 800 964 5118 or +1 781 641 5118 15
vol. VI, no. 9
of estimates. This way we can adjust our Park, Robert. Checklists and Criteria for
input parameters in order to compute not Evaluating the Cost and Schedule Estimating
only an answer about duration, effort, qual- Capabilities of Software Organizations. SEI-
ity, and features, but also a risk level that the 95-SR-005, January 1995.
organization can tolerate. Without the range
Park, Robert. Goal-Driven Software
and risk level, it’s all or nothing — not a
Measurement — A Guidebook. SEI-96-
very rational, or realistic, approach. How do
HBK-002, August 1996.
we know on a point estimate what is the
probability of achieving it? Park, Robert et al. Software Cost and
Schedule Estimating: A Process Improvement
Initiative. SEI-94-SR-03, May 1994.
Conclusion
Whether or not you use my list of criteria, be Web Sites
sure to list your own before you go shopping
www.incose.org/tools/tooltax/costest_tools.
for a macro-estimation tool. Tool vendors
html. Provides an interesting, though
have a technical name for those who seek
slightly out of date, list of software cost
tools without stated, written criteria: loose
estimating tools.
wallets!
www.methods-tools.com/tools/frames_
projmgmt.html. Offers a more up-to-date
Acknowledgements
list, containing many other tools.
This article is based on a presentation as
moderator of the January Maryland Society
for Software Quality Roundtable 2000, About the Author
“Help! for estimation.” Stan Rifkin is a principal with Master
Systems Inc., a firm that offers advisory
services related to software improvement.
References
He worked at the Software Engineering
The following SEI references are often avail-
Institute on software process improvement,
able for free download on the SEI Web site:
the American Association for the Advance-
www.sei.cmu.edu/publications/.
ment of Science as CIO, and the National
McAndrews, Donald. Establishing a Headquarters of the American Red Cross
Software Measurement Process. SEI-93-TR- as the director of systems development.
16, July 1993. Mr. Rifkin’s previous articles have appeared
in March and May 2000 issues of ITMS.
Park, Robert. A Manager’s Checklist for
Mr. Rifkin can be reached at sr@
Validating Software Cost and Schedule
Master-Systems.com.
Estimates. SEI-95-SR-004, January 1995.

q Please start my subscription to IT Metrics StrategiesÒ for one year at $485, or US $545 outside North America.
Phone Megan Nields at +1 781 641 5118, or fax +1 781 648 1950, or e-mail info@cutter.com.
q Please renew my subscription.

Name q Payment or purchase order enclosed


Title q Please bill my organization
q Charge my Mastercard, Visa, American Express,
Organization Diners Club, or Carte Blanche
220*5ITS
Dept. Card no.

Address Expiration Date

City State/Province Signature

Zip/Postal Code Country Web site: www.cutter.com/itms/


Cutter Information Corp.
Tel. Fax
Suite 1, 37 Broadway
E-mail Arlington, MA 02474-5552 USA

You might also like