Inforr?iQrion nnd so&W?

Technology

1995 37 (2) 113-I 18

When to prototype: decision

variables used in industry
Bill C Hardgrave
Computer Information Systems and Quantitative Analysis, College of Business Administration, University of
Arkansas, Fayetteville, AR 72701, US
internet:whardgra@comp.uark.edu

Prototyping continues to gain acceptance as a way to improve the development of information

systems. Recent surveys indicate over 60% usage in industry. However, there exists widespread
disagreement concerning the proper usage of prototyping. Speci&ally, the question is: When
should prototyping be used ? This study is an attempt to answer this question by examining the
factors used by IS managers in their decision to use prototyping. Results of the study indicate
a set of variables used by practitioners. Thus, it appears that, based upon the results of this
study and similar studies, a set of factors is being used by IS managers in their decision to
prototype.
Keywords:

prototyping,

systems development,

decision

variables

During the early 1980s prototyping emerged as a potential

solution to the problems facing information systems
development. With the promise of improved systems, faster
development, and higher user satisfaction, prototyping
quickly gained acceptance. The use of prototyping has
continued to grow during the past several years. A 1984
industry survey by Langle et al., indicated that 33 % of
the respondents use prototyping. In 1987, 46% usage was
reported*; 49% in 19883; and 61% in 19904.
However, if prototyping does indeed provide all the
purported advantages, then why is the reported adoption of
prototyping not closer to lOO%? The reason is simple: if
not used properly, prototyping can be counterproductive.
Potential disadvantages of prototyping include: poor
documentation, uncontrollable projects and unrealistic
expectations from users, among others. Industry users
quickly realized that prototyping was not a remedy for all
problems associated with systems development.
Because prototyping is not a cure-all, IS managers must
make the decision whether or not to use prototyping for a
specific project. Although the IS literature identifies several
variables which can assist the manager in his/her decision,
most of the variables have not been empirically
investigated. Therefore, the purpose of this study is to
identify empirically those decision variables which are
being used in industry. This is accomplished by an industry
survey which asks IS managers to identify those variables
which are used to make the decision whether or not to
prototype.

0950-5849/95/$09.50

0 1995 Elsevier Science B.V. All rights reserved

Background
Although prototyping has been used for several years, there
appears to be no unique definition, nor an established set of
guidelines (decision variables) for its use. An investigation
of the literature reveals a wide variety of variables which
allegedly impact the decision to prototype.
In an effort to determine, among other things, the benefits
and disadvantages of prototyping, Carey and Currey asked
questionnaire respondents to indicate when they thought
prototyping was the best strategy. Although this is not a
direct indication of the decision variables used in the
prototyping decision, it does provide some of the reasons
prototyping was initially considered. Their findings
indicated that prototyping was the best strategy for: (a) all
on-line development; (b) all new development; (c) when
user expectations or requirements are not clear; and (d)
when the appropriate tools are available.
Doke et aL6, used a Delphi panel of industry experts
to indicate the factors used in their decision to use
prototyping. The top factors determined from the study
were:
communicate design widely;
on-line system;
reduce development time;
developers lack experience;
feasibility in question;
alternatives need evaluated;
large and complex system;

113

When to prototype: B C Hardgrave

. early results needed;

. need user participation;
and
. unclear requirements and expectations.
A study by Hardgrave and Wilson7 identified 18 guidelines from the IS literature which suggest the proper usage
of prototyping. For example, one of the guidelines is: If
the system mode is on-line, then prototyping should be
used. However, of the 18 guidelines, very few have been
empirically tested. Instead, most are based upon supposition or pure speculation7.
Additionally,
many of the
guidelines are contradictory
(both within and between
guidelines), which indicates great disparity among those
espousing the decision variables. The decision variables
identified by Hardgrave and Wilson7, derived from the
literature, include:
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

clarity of requirements;
requirements stability;
system mode;
project duration;
innovation;
project size;
project impact;
performance requirements;
user involvement;
users experience with prototyping;
users experience with computers;
number of users;
impact on users;
developers familiarity with application domain;
developers experience with prototyping;
management support;
project feasibility; and
tool availability.

Each of the aforementioned

studies has its limitations.
Hardgrave and Wilsons7 study indicates that most of the
guidelines (i.e., decision variables) are not empirically
tested. Carey and Curreys study was not designed to
capture the decision variables, although a list of variables
which could be considered was determined. Finally, Doke
et ~1.~ used a Delphi panel of only 16 members, which
greatly limits the generalizability
of their findings. Thus, a
need exists for a collection of the prototyping decision
variables as used by industry firms.

Table 1. Respondent
Organization

profile

characteristics

Total number of employees

Total number of IS personnel
Respondent

Average
2810
67

Minimum
7
I

Maximum
22 000
800

Average
15

Minimum
3

Maximum
31

characteristics

Years of IS experience
Experience with prototyping
(1 =none; 7 =extensive)
Industry
Manufacturing
and service
Diversified financial
Insurance
Retail
Transportation
Utilities
Education
Health Service
Government
Other

43%
2%
4%
8%
4%
4%
8%
4%
20%
2%

Does not equal 100% due to rounding

The remaining 29% of the respondents had never used

prototyping and would not have the knowledge needed to
answer the question.
Of those using prototyping,
63
provided information regarding the decision variables used
in making the decision to prototype. Thus, 75% (63 of 84)
of the respondents consciously used decision variables in
the selection of a prototype strategy (rather than applying
prototyping to all projects). The 63 respondents represent
a diverse group of companies, as shown in Table 1.
As noted above, respondents were asked to list and rate,
rather than rank, each factor (i.e., decision variable).
Rating is more appropriate in those cases where the factors
are not provided u priori. Also, rating requires less mental
effort because the factors can be evaluated one at a time
rather than requiring simultaneous
consideration
of all
factors. According to Niederman et aL8: Rating allows
respondents to shpw indifference among issues (by giving
them the same rating) and also allows them to show relative
strength of judgement (by using a wide or narrow range of
ranking assignments). Respondents were free to identify as
many factors as they wanted, but no respondent identified
more than 10 factors.

Results
Data collection
A mail survey was used to collect data from 118
Information Systems (IS) managers throughout the United
States. Of the 118 respondents, 84 used prototyping (7 1% ) .
For those firms using prototyping,
the IS manager was
asked to complete the following:
Please list the factors that you feel should be
considered in the decison whether or not to use
prototyping
for a specific system development
project. For each factor listed, indicate the importance of the factor on a scale from 1 to 10 (1 = low
importance; 10 = high importance).

114

The 63 respondents provided a list of 231 factors used in

the prototyping decision (an average of almost four factors
per respondent). Although many of the respondents used
the same terminology
(for example, complex system),
most of the terms were slightly different. Therefore,
some editing of the responses was required to eliminate
redundancy, provide a shorter list of variables and ensure
consistent terminology. For example, the terms ambiguous
requirements and users dont know what they want were
combined under the heading requirements determination.
By editing, the original list of 231 variables was reduced
to 48.
Many of the variables were listed by only one or a few of

Information and Sojiware Technology 199.5 Volume 37 Number 2

When to prototype: B C Hardgrave

Table 2. De&on variabks by frequency

Frototypingde&ion

Frequency

Sum

Mean

VariaMW

Std

Range

DW

importance. Thus, frequency of response and the importance of that factor provide different views of the decision
factors. A detailed discussion of the 12 factors is provided
in the next section.

Reouirements
dktermination
Project size
Complexity
Availability of tools
Mode

24
19
16
16
16

200
139
127
123
128

8.33
7.32
7.94
7.69
8.00

2.13
2.85
2.56
2.23
2.18

3-10
l-10
l-10
3-10
3-10

Discussion

Project duration
Feasibility/testing
Innovativeness
User exuerience
User in;olvement
Impact (importance)
Developer experience

15
14
14
12
12
10
8

110
115
104
78
105
75
54

7.33
8.21
7.43
6.50
8.75
7.50
6.75

2.55
2.14
2.38
2.60
1.30
2.91
2.99

2-10
4-10
3-10
3-10
6-10
2-10
l-10

Requirements

Table sorted by frequency (i.e., number of times listed by respondents)

Table 3. Decision variables by mean

Prototyping decision
variables

Frequency

Sum

Mean

STD

Range

DW

User involvement

12

105

8.75

1.30

6-10

Requirements
Feasibility/testing
determination

24
14

200
115

8.33
8.21

2.13
2.14

3-10
4-10

Mode

16

128

8.00

2.18

3-10

Complexity

16

127

7.94

2.56

l-10

Availability of tools
Impact (importance)
Innovativeness
Project duration
Project size
Developer experience
User experience

16
10
14
15
19
8
12

123
75
104
110
139
54
78

7.69
7.50
7.43
7.33
7.32
6.75
6.50

2.23
2.91
2.38
2.55
2.85
2.99
2.60

3-10
2-10
3-10
2-10
l-10
l-10
3-10

Table sorted bv mean reswnse

the respondents. For example, system throughput was

identified by only one respondent. Because it is not possible
to provide a full discussion of all 48 factors in an article of
reasonable length, only the top variables will be discussed
further. Tables 2 and 3 provide a summary of the top 12
decision variables identified by the respondents. To be
included in the tables, at least 10% of the respondents must
have identified the factor (i.e., at least seven respondents).
Although the inclusion of only 12 factors for discussion is
arbitrary, it is not meant to diminish the importance of the
remaining 36 factors. The list of all 48 variables can be
found in the Appendix.
Table 2 lists the factors in order of the number of times
identifed by the respondents. For example, requirements
determination was identified by 24 respondents. The
frequency of response indicates the popularity or breadth of
use of the factor in making the prototyping decision, but it
does not indicate the importance of the factor.
Table 3 lists the factors in order by the mean response.
For example, the mean response for requirements
determination is 8.33 (1 = low importance, 10 = high
importance). The ranking by mean response indicates the
importance of the factor in the prototyping decision. Both
tables are provided because the rankings are different
between Tables 2 and 3. For example, user involvement
is ranked tenth by frequency, but first when ordered by

Information

and S&ware

Technology 1995 Volume 37 Number 2

The discussion in this section will follow the order of the

decision variables as presented in Table 2.
determination

Respondents indicated that prototyping should be used under

conditions of ambiguous requirements, requirements
expected to change, users dont know what they want,
expected user modifications and process not well defined.
Perhaps the greatest benefit provided by prototyping is the
ability to determine system requirements when requirements
are particularly ambiguous, the users do not know exactly
what they want, or the requirements are expected to change.
Prototvning captures an initial set of reauirements and.
through it&at&e discovery, builds the iystem to meet
the users needs.: 38% of the respondents indicated
requirements determination to be a factor in the decision
to prototype.
Project size

Project size refers to the man-hours and/or cost necessary

to produce the system. Project size does not include project duration or number of users; 30% of the respondents
indicated that the size of the project helped determine
whether or not prototyping should be used. The argument
for prototyping large systems is that, because of its size,
specifications will change during the development of the
system3X4.Prototyping readily accommodates change,
thus proving its usefulness.
Complexity

Several respondents simply specified that complexity was

a factor in their choice to use prototyping. Because no
explanation of complexity was provided, it is difficult to
determine exactly what the respondent meant. However,
one would expect complexity to be determined by such
characteristics as: age of technology, number of users,
project size, volume of new information generated by
system, and complexity of producing new information9*5.
Burns and Dennis suggest that prototyping is not suitable
for projects of high complexity because prototyping lacks
the discipline and structure needed to manage and control
the project.
Availability

of tools

Responses such as ability to convert prototype to final

product, ease of prototyping, ease of use of tools, and
methodology/tool match indicate that the availability of
tools must be considered in the prototyping decision.
However, the type of tool needed is dependent upon the
type of prototyping used.
Expendable prototypes, which are discarded after they
are no longer needed, can be created with simple tools,

115

When to prototype: B C Hardgrave

such as word processors or graphics packages (e.g.,

Microsofts Powerpoint)637. Evolutionary prototypes,
which evolve into the final operational system, require
more sophisticated tools, such as database management
systems, fourth generation languages, or special-purpose
prototyping tools. These tools are needed to provide fast
response to user requests and to allow the prototypes to
evolve into the final project9X9.
Respondents identifed tools such as EIS Toolkit,
Excelerator, IEF, and numerous others, as aids to facilitate
prototyping. It would appear that almost any company
possesses the tools needed for expendable prototyping
(which could be hand-drawn on paper, if necessary), but
may not have the necessary tools needed for evolutionary
prototyping.
Mode

System mode can be categorized as on-line, batch, or a

mixture (of on-line and batch activities). Respondents
indicate that systems which contained some elements of online processing should employ a prototyping approach. Online systems are much more related to user needs and
expectations than batch systems (because of the user
interface) and a prototyping approach should be u~ed~,~.
Project duration

Project duration, the time between the start of a project and

the delivery of the final system to the user, is an important
consideration in the selection of a development strategy.
Long-running projects encounter many problems caused by
organizational and individual changes as a direct
consequence of the passage of time. The longer the duration
of a project, the greater the changes are likely to be.
Because of its iterative nature, prototyping can facilitate
these changes2*.
Feasibility/testing

Management is often unwilling to experiment with new

concepts, ideas, and procedures, due to the risk level of
such undertakings23. A good project may sometimes be
dismissed because management will not commit resources
before fully knowing the benefits. Prototyping can be used
in situations where there is a need for experimentation and
learning before commitment of resources for a full
project24,25.The prototype provides the opportunity to test,
modify, and visualize a real-life system without tbe
resources needed for the full project. Prototyping allows
management to evaluate the system and make an informed
decision. Some of the terms used by respondents to
describe this include: test some of the concepts,
validation of concepts, find out what system can do, and
determine impact on database.

systems (low innovation). Thus, it would seem appropriate

to use prototyping, which facilitates the interaction between
user and developer22. Respondents indicated that systems
considered new development, from scratch, or automation of manual process should use a prototyping
approach.
User experience

User experience refers to the users exposure to

computers and computerized systems. Lack of automation
experience is seen as a risk induce?. In this case, prototyping is a way to decrease risp. Users unfamiliar with
automated systems can develop a better idea of their system
requirements by being exposed to a prototype3,2s. Sophistication of user, computer literacy of user, and users
DP experience, were considered by this studys respondents
in the decision to use prototyping.
User involvement

Almost all advocates of prototyping point to user

involvement as one of the major advantages of prototyping.
Prototyping, compared to the traditional systems development life cycle, requires more communication and interaction between the user and developer. Respondents felt
that, when user involvement was necessary, prototyping
should be used to facilitate the involvement. For example,
some indicated that prototyping should be used to gain user
interest and increase user activity in development
process.
Impact (importance)

Responses such as criticality of system, mission critical

status, effect on business, and potential harm of bad
system indicate that IS managers view the impact of the
system on the organization as a factor in making the prototyping decision. Critical systems-systems that operate,
manage, and control the daily business activities-have a
very broad and strong impact on the organization.
Prototyping should be used for critical systems22329.For
critical systems it is extremely important that the system
meet specifications-prototyping
facilitates this important
requirement.
Developer experience

A developers experience with similar applications and the

developers overall experience impact the amount of
required user interaction. Possibly, if a developer has
development experience with similar applications (i.e.,
high developer-task proficiency), then prototyping should
not be used3. The reason is that when the task is familiar
to the developer, less user interaction is required to
determine requirements. Unnecessary involvement of users
may increase development time3.

Innovativeness

Innovation is an indicator of tbe foundation of the project,

i.e., is it a new development (high innovation) or a
modification to an existing system (low innovation)? New
system development (high innovation) requires a higher
user/developer interaction than modifications to existing

116

Other variables worth mentioning

Other variables that respondents indicated, but did not

meet the 10% cut-off for inclusion in the table, include:
developer experience with prototyping tool (six times);
management support (four times); history of user/developer

Information

and Sojiware

Technology 1995 Volume 37 Number 2

When to prototype: B C Hardgrave

Table 4. Decision variables:

a comparison

of this studys findings to

previousstudtes
Decisionvariables

This study

Doke et al. Carey and

(1992)
C-Y
(1989)

Unclear requirements
Large systems
Complex systems
Availability of tools
On-line systems
Project duration
Feasibility/testing
New development
User lacks DP experience
Need user involvement
Critical system
Developers experience
Need early results
Alternatives need evaluated
Communicate design widely

X
X
X
X
X
X
X
X
X
X
X
X

X
X
X
X
X
X

X
X

when the user lacks DP experience? Additional research is

needed to evaluate the impact of the variables and the
resulting decision to use prototyping on the eventual
success of the information system.

Acknowledgements
The author is indebted to the reviewers for their valuable
comments, and to the companies that participated in this
study.

X
X

References

X
X
X
X

communication (three times); and number of users (two

times). According to the responses received, prototyping
should be used when the developer has experience with the
prototyping tool, when the project has management support,
and when only a few users are involved. Additionally,
prototyping may facilitate better communication when a
history
of poor communication between users and developers
exists.

Summary
Table 4 summarizes the previous discussion and compares
the results of this study to the empirical studies mentioned
in the second section. As seen in Table 4, all four of the
variables in Carey and Curreys study were also
uncovered in this study. Similarly, eight of the variables
from the Doke et aL6 study corresponded to variables
from this study. Of the top 12 variables discussed in this
paper, only two variables, user lacks DP experience and
critical system, were not included in either the Carey and
Currey or Doke et al. 6 study. The close association among
the three studies suggests that a set of decision variables
does indeed exist and is being used in industry.

10
II

12
I3

I4
15

Conclusion
I6

This study has attempted to determine a set of variables

used by IS managers in their decision to use a prototyping
approach to IS development. The top 12 variables which
were suggested by respondents include: unclear requirements; large systems; complex systems; availability of
tools; on-line systems; project duration; feasibility/testing;
new development; user lacks DP experience; need user
involvement; critical system; and developers experience.
These variables correspond closely with similar studies
from Carey and Currey5 and Doke et aL6.
It appears that, based upon the results of this study and
similar studies, a set of factors is being used by IS managers
in their decision to use prototyping. Future research must
now evaluate the quality of these variables. For example:
Is prototyping the best strategy to use when the requirements are unclear? Is prototyping the best strategy to use

hformation and Soj?ware Technology I995 Volume 37 Number 2

I7

I8
I9

20
21

22
23
24

Langle, G B, Leitheiser, R L and Naumann, J D A survey of

applications systems prototyping in industry In& & Manage. Vol 7
(1984) pp 273-284
Necco, C R, Gordon, C L and Tsai, N W Systems analysis and
design: current practices, MIS Quarterly Vol 11 No 4 (December
1987) pp 461-475
Carey, J M and McLeod, R Use of system development methodology
and tools J. Syst. Manage. VoI 39 No 3 (March 1988) pp 30-35
Doke, E R An industry survey of emerging prototyping methodologies
In$ & Manage. Vol I8 (April 1990) pp 169-176
Carey, J M and Currey, J D The prototyping conundrum Datamation
Vol 35 No 11 (June 1989) pp 29-33
Doke, E R, Swanson, N E and Hardgrave,
B C The decision to
prototype information systems: a pilot study 1992 Proc. of the National
Decision Sciences Institute, San Francisco, CA (November 1992) pp
917-919
Hardgrave,
B C and Wilson, R L A survey of guidelines for the
proper usage of information system prototyping
1993 Proc. of the
National Decision Sciences Institute, Washington,
DC (November
1993) pp 830-832
Niederman,
F, Brancheau, J C and Wetherbe, J C Information
systems management issues for the 1990s MIS QuarterlyVol 15 No 4
(December 1991) pp 475-500
Bums, R N and Dennis, A R Selecting the appropriate application
development methodology Dorabase Vol 17 No 1 (Fall 198.5) pp 19-23
Davis, G B Strategies for information requirements determination
IBM Systems Journal Vol 21 (1982) pp 4-3 I
Naumann, J D and Jenkins, A M Prototyping: the new paradigm for
systems development MIS Quarterly Vol6 No 3 (September 1982) pp
29-44
Saarinen, T System development methodology and project success
If. & Manage. Vol 19 (1990) pp 183-193
Deamley, P A and Mayhew, P J In favour of system prototypes and
their integration into the systems development cycle Computer J. Vol
26 No 1 (1983) pp 36-42
Guimaraes, T Understanding implementation failure J. Syst. Manage.
Vol 32 No 3 (March 1981) pp 12-17
El Louadi, M, Pollalis, Y A and Teng, J T C Selecting a systems
development
methodology:
a contingency
framework
In5 Res.
Manage. J. Vol 4 No 1 (Winter 1991) pp ll--19
Burch, J G Systems analysis, design, and implementation Boyd &
Fraser, Boston, MA (1992)
Carey, T T and Mason, R E A Information system prototyping:
techniques, tools, and methodologies ZNFOR Vol 21 (August 1983)
pp 177-191
Whitten, J L, Bentley, L D and Barlow, V M Systems analysis &
design methods (3rd edn) IRWIN, Burr Ridge, IL (1994)
Alavi, M An assessment of the prototyping approach to information
systems development
Comm. ACM Vol 27 No 6 (June 1984) pp
556-563
Carey, J M Prototyping: alternative systems development methodology If: and Soft. Technol. Vol32 No 2 (March 1990) pp 119-126
Graham, D R Incremental development:
review of nonmonolithic
life-cycle development models If: and Sof. Technol. Vol 31 No 1
(January/February
1989) pp 7-20
DOS Santos, B L MIS project mangement: a contingency approach
Management Decision Vol 26 No 6 (1988) pp 22-28
Pressman, R S Sojiware engineering: a practitioners approach (3rd
edn) McGraw-Hill (1992)
Alavi, M The evolution
of information
systems development
approach: some field observations DATABASE Vol 15 No 3 (Spring
1984) pp 19-24

117

When to prototype: B C Hardgrave

25 Mahmood,
M A System development
methods-a
comparative
investigation
MIS Quarterly Vol 11 No 3 (September 1987) pp
293-311
26 Ahituv, N, Hadass, M and Neumann, S A flexible approach to
information system development MIS Quarterly Vol 8 No 2 (June
1984) pp 69-78
27 Tate, G and Verner, J Case study of risk management, incremental
development, and evolutionary prototyping In5 and Soft. Technol.

Vol 32 No 3 (April 1990) pp 207-214

28 Gavurin, S L Where does prototyping fit into IS development? J.
Sysf. Manage. Vol 42 No 2 (February 1991) pp 13-17
29 Boar, B Application prototyping: a life cycle perspective J. Syst.
Manage. Vol 37 No 2 (February 1986) pp 25-31
30 Kraushaar,
J M and Shirland, L E A prototyping
method for
applications
development
by end users and information
systems
specialists MIS Quarterly Vol 9 No 3 (September 1985) pp 189-197

Appendix: Complete list of factors

(in order by frequency of response)
Requirements determination (24)
Project size (19)
Complexity (16)
Availability of tools (16)
Mode (16)
Project duration (15)
Feasibility/testing (14)
Innovativeness (14)
User experience (12)
User involvement ( 12)
Impact (importance) (10)
Developer experience (8)
Developer experience with tools (6)
Management support (4)
History of end-user/systems analyst communication (3)
Reusability (3)
Impact on other programs (3)
User satisfaction (2)
Application platform (2)
Training tool (2)
Number of users (2)
Manhours available (2)
Distributed or centralized (1)
Design definition (1)

Type of project (1)

Interference with user needs (1)
Long life requirements (1)
Project definition (1)
Provide sign-off document (1)
Identify data elements (1)
Relational DBMS design (1)
Information flow (1)
Need to manage expectations (1)
Impact on users (1)
Request (1)
Program design (1)
Short life requirements (1)
System throughput (1)
User availability (1)
Data structure (1)
Consistency (1)
Conference room pilot (1)
Broad brush specifications (1)
Number of team members (1)
Cross applications (1)
Prototyping on small scale (1)
System flow-keying information (1)
Human engineering (1)

Note: the number in parentheses indicates the frequency of response.

118

Information and Sofrware Technology 1995 Volume 37 Number 2