EJISE Volume 11 Issue 3

B2C e-Commerce Success: a Test and Validation of a Revised
Conceptual Model
Irwin Brown and Ruwanga Jayakody
University of Cape Town, South Africa
Irwin.Brown@uct.ac.za
Abstract: Since the advent of the Internet, B2C e-Commerce has grown substantially across the globe. Whilst much
research has examined factors influencing adoption of e-commerce, not as many studies have investigated the postadoption phenomenon of success. Those studies that have investigated IS success and the extensions required to
accommodate e-commerce have mainly been conceptual. Few have attempted to test and validate the models
empirically. The purpose of this study was to fill this gap. By drawing from the technology acceptance model,
expectation-confirmation theory and IS success theory, a revised conceptual model was derived. The model and
relationships were tested and validated using data gathered from 166 online consumers in South Africa. 7 interrelated
dimensions of B2C e-commerce success were confirmed, namely service quality, system quality, information quality,
trust, perceived usefulness, user satisfaction and continuance intentions. Direct relationships between dimensions were
identified. These showed that user intentions to continue using an online retail site are directly influenced by perceived
usefulness, user satisfaction and system quality. User satisfaction is directly influenced by service quality and perceived
usefulness, whilst perceived usefulness is directly influenced by trust and information quality. Trust in the online retailer is
directly influenced by service quality and system quality. The implications of these and other findings are discussed.
Keywords: IS success; e-commerce success; B2C e-commerce; DeLone and McLean
1. Introduction
E-commerce is generally accepted as being a sub-set of e-business (Kim et al., 2006; Pavic et al., 2007). Ebusiness activities and applications range from simple email to e-enabled supply chain management (Fusilier
& Durlabhji, 2003; Parker & Castleman, 2007). E-commerce on the other hand is more narrowly defined in
terms of the purchasing and selling products or services through the medium of the Internet (Grandon &
Pearson, 2004). Business-to-consumer (B2C) e-commerce, the focus of this paper, refers specifically to the
activity in which consumers buy products or services using the Internet medium (Pavlou & Fygenson, 2006).
Despite the dot-com failures of the 2000s, B2C e-commerce has continued to grow steadily (Van Slyke et al.,
2004). This growth has occurred not only in the developed countries, but in the developing as well, resulting
in an increasingly global community of online shoppers (Cyr et al., 2004). South Africa is no exception in this
regard. Goldstuck (2002, 2004) and World Wide Worx (2006) have reported consistent increases in South
African online retail sales revenue (excluding airline tickets) over the past few years, rising from R 162 million
in 2001 to R 514 million in 2005. This is indicative of a relatively healthy B2C e-commerce environment,
which is even healthier if sales of airline tickets are also considered (World Wide Worx, 2006). Companies
are making large investments in e-commerce applications but are hard pressed to evaluate the success of
their e-commerce systems (DeLone & McLean, 2003). This growth in e-commerce is the reason behind
recent attempts to measure the success of e-commerce.
Much of the discussion on B2C e-commerce success has been based on earlier research on IS success by
DeLone & McLean (1992). An example of this is the e-commerce success model developed by Molla &
Licker (2001). Grounding conceptualizations of e-commerce success in IS literature is perfectly valid given
that for all intents and purposes B2C e-commerce systems are information systems that have been extended
for direct use by consumers (DeLone & McLean, 2004; Garrity et al., 2005; Pather et al., 2004). There is
often a lack of conceptual clarity as to the theoretical basis for relationships between dimensions of B2C ecommerce success this being as a result of the conceptual weaknesses in the foundational IS success
models (Seddon, 1997). Establishing the basis for these relationships and clarifying their nature is therefore
an important endeavour (DeLone & McLean, 2003, 2004). There have been surprisingly few attempts at
empirically validating and testing models of B2C e-commerce success such as that developed by Molla &
Licker (2001) and DeLone & McLean (2003).
Alternative e-commerce success models have been suggested. Torkzadeh & Dillhon (2002) identified a set
of success factors for e-commerce. Unlike Molla & Licker (2001) and DeLone & Mclean (2003), no
relationships between factors were identified. Garrity et al. (2005) too developed an e-commerce success
model, but this tended to view success primarily in terms of dimensions of satisfaction. Quaddus & Achjari
ISSN 1566-6379
167
Academic Conferences Ltd
Reference this paper as:
Brown, I. and Jayakody, R. B2C e-Commerce Success: a Test and Validation of a Revised Conceptual Model. The
Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 167 - 184, available online at
www.ejise.com
Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)
(2005) developed an e-commerce success framework that reflected an organizational perspective. In this
study, a customer perspective was being sought, and so the DeLone & McLean (2003) model and its
variants were deemed to provide a more appropriate base.
The overall purpose of the study was to test and validate a revised conceptual model of B2C e-commerce
success. The detailed objectives were to:
Re-examine the relationships between key dimensions of B2C e-commerce success in the light of
established theories, and to develop a revised model;
Empirically validate and test the model using data gathered from a sample of online consumers in
South Africa;
Add to the body of knowledge on B2C e-commerce in South Africa. Whilst there have been several
major South African academic studies examining e-commerce success from an organizational
perspective (Molla, 2002; Pather et al., 2006), few have investigated the consumer perspective. An
exception is De Villiers & Van Der Merwe (2001). This study will build on the work of De Villiers & Van
Der Merwe (2001).
In the next section, the conceptual background of the study is laid out and the research model is developed.
The research methodology is then outlined. The data analysis and results are reported before these results
are discussed and implications drawn. The limitations of the study are noted and ideas for further research
put forward. Finally the paper is concluded.
2. Conceptual background
Theory in the field of IS success has stemmed primarily from the seminal work of DeLone & McLean (1992).
Numerous studies have since sought to extend and/or validate this framework. A summary of some key
studies and variables examined are shown in Table 1 below. The table shows 11 dimensions of IS success
that have been identified. The dimensions of system quality, information quality and user satisfaction have
been included in all of the studies. This indicates that they are central to measuring IS success and should
be included in any success model.
Seddon & Kiew (1996) attempted to partially validate the DeLone & McLean (1992) model and in so doing
suggested the inclusion of perceived usefulness as a replacement to use, given that use is primarily a
behaviour, and not reflective of success in contexts where usage is mandatory.
Individual impact and organizational impact were included in the original DeLone & McLean (1992) model,
but not in most subsequent studies. DeLone & McLean (2003) in a 10-year update of their 1992 IS success
model suggested that individual, organizational and other impacts be collapsed into a single net benefits
construct for the sake of parsimony.
Rai et al. (2002) empirically tested and validated both the DeLone & McLean (1992) and Seddon (1997)
model, but excluded the individual impact and organizational impact dimensions. Iivari (2005) also attempted
to validate the DeLone & McLean (1992) model. In so doing he demonstrated that individual impact could be
assessed with a perceived usefulness measure. He did not include organizational impact as a dimension.
Molla & Licker (2001) conceptualized a model of e-commerce success by drawing from the work of DeLone
& McLean (1992). In addition to information (or content) quality, system quality, use and user (e-commerce
customer) satisfaction, they also highlighted support and service (service quality) and trust as additional
factors to consider in a B2C e-commerce environment. They noted too that in a B2C e-commerce
environment usage of a system is typically voluntary, unlike in work situations where usage of a system may
be mandatory.
DeLone & McLean (2003) in their updated model included the core dimensions of information quality, system
quality and user satisfaction, as well as use/intentions to use, net benefits and service quality. They
suggested that intention to use may be employed as an alternative to use as a success dimension. DeLone
& McLean (2004) illustrated how the updated model could without modification be used to evaluate ecommerce success.
www.ejise.com
168
Table 1: Dimensions of IS Success

Dimension
D&M
(1992)
X
X
X
X
X
X
S&K
(1996)
X
X
X
Seddon
(1997)
X
X
X
X
X
X
X
X
M&L
(2001)
X
X
X
X
System Quality
Information Quality
User Satisfaction
Use
Individual Impact
Organisational Impact
Perceived Usefulness
X
Net Benefits of Use
Service Quality
X
Trust
X
Intention to Use
Key: D&M DeLone & McLean; S&K: Seddon & Kiew; M&L: Molla & Licker
Rai et al.
(2002)
X
X
X
X
D&M
(2003/4)
X
X
X
X
Iivari
(2005)
X
X
X
X
X
[X]
X
X
X
Intention to use is primarily a pre-adoption construct (Davis, 1989). It gives an indication of users intentions
concerning future use of an information system. It is thus not entirely suitable for post-adoption studies,
where attempts are made to evaluate the success of an information system (Bhattacherjee, 2001).
Bhattacherjee (2001) makes the case for assessing user intentions to continue using a system (continuance
intention). Such a measure is more suited to post-adoption studies such as the investigation of IS success.
This construct is furthermore appropriate as a success measure in the e-commerce context, where repeat
customers are highly valued. Bhattacherjee (2001) found that perceived usefulness, user satisfaction and
loyalty incentives were influences on continuance intention. In the next section, a research model is
developed by drawing out the salient dimensions of e-commerce success and the relationships between
them.
3. Research model
Based on the discussion in the conceptual background section above, the factors that are important in the
context of B2C e-commerce systems include the core IS success measures of system quality, information
quality (also referred to as content quality in e-commerce environments) and user satisfaction (also referred
to as customer satisfaction in e-commerce environments) (Molla & Licker, 2001). Additional factors relevant
to e-commerce specifically are trust, service quality (or support and service), loyalty incentives and
continuance intention (Bhattacherjee, 2001; Molla & Licker, 2001). As suggested by DeLone & McLean
(2003) [continuance] intention was chosen as an alternative to use, given the difficulties of measuring
general e-commerce use. Use therefore was not included in our research model. For the same reason net
benefits was not included in our research model, but rather perceived usefulness, which is a valid, easily
assessable perceptual measure of net benefits (Iivari, 2005; Rai et al., 2002; Seddon, 1997). The
relationships between these 8 chosen factors will now be discussed by examining each in turn.
3.1 Continuance intention

Research into IS success has been concerned with the quest for the dependent variable (DeLone & McLean,
1992). DeLone & McLean (1992) initially proposed the organizational impact of an IS as the ultimate
dependent variable, preceded by individual impact. In the 10-year update, DeLone & McLean (2003)
combined these into a net benefits construct and argued for this as the dependent variable. Given the
difficulties associated with assessing usage and net benefits of use as success measures, continuance
intention provides an appealing yet credible alternative as the ultimate dependent variable. The focus in this
study is on how the various independent variables ultimately impact directly and indirectly on it. As a
consequence only unidirectional relationships are considered. Consistent with many mainstream studies in
IS, null hypotheses are not expressed (Bhattacherjee, 2001; Gefen et al., 2003; Iivari, 2005).
3.2 User satisfaction

User Satisfaction is the most general perceptual measure of information systems success (Seddon, 1997). In
the e-commerce environment it is an important means of measuring customers opinions of the e-commerce
system (DeLone & McLean, 2003). Customers in this study refer to only those who directly use e-commerce
services. Wang et al. (2001) developed a measure of customer information satisfaction with e-commerce
websites. The measure had dimensions of customer support, security, ease of use, transactions and
payment, information content, digital products and services and innovation. Several of these dimensions
www.ejise.com
169
ISSN 1566-6379
overlap with those in IS success models (DeLone & McLean, 2003; Molla & Licker, 2001). For example
customer support (service quality), ease of use (system quality) and information content (information quality)
appear as dimensions in both success and satisfaction measures. This indicates a lack of conceptual clarity
between satisfaction and success in the IS literature (Garrity et al., 2005). As a consequence many studies
that incorporate user satisfaction into IS success models employ single item measures for this construct (Rai
et al., 2002). Bhattacherjee (2001) identified user satisfaction as a key determinant of whether a user intends
to continue using an e-commerce system. The hypothesis suggested is:
H1: User Satisfaction has a positive effect on Continuance Intention for an e-commerce system.
3.3 Perceived usefulness

Rai et al. (2002) in validating the Seddon (1997) model demonstrate that perceived usefulness positively
influences user satisfaction with an information system. The hypothesis suggested is:
H2: Perceived Usefulness has a positive effect on User Satisfaction for an e-commerce system.
Bhattacherjee (2001) demonstrated perceived usefulness as a key determinant influencing customer
intentions to continue using an e-commerce system, justifying the following hypothesis:
H3: Perceived Usefulness has a positive effect on Continuance Intention for an e-commerce system.
3.4 Loyalty incentives

Bhattacherjee (2001) drew on expectation-confirmation theory (ECT) in the consumer behavior literature,
technology acceptance model (TAM) in the IS use literature and agency theory in the organizational
economics literature to propose satisfaction, perceived usefulness and loyalty incentives as the three key
factors influencing customers decisions to continue using an e-commerce system. Loyalty incentives has not
previously been included in IS success models, but based on Bhattacherjee (2001) it is hypothesized that:
H4: Loyalty Incentives have a positive effect on Continuance Intention for an e-commerce system.
3.5 Trust
Gefen et al. (2003) note that there should be a clear distinction between trusting beliefs and trusting
behaviours (e.g., usage of an e-commerce system). In this paper trust is viewed as a set of trusting beliefs
about e-commerce (Gefen et al., 2003). Molla & Licker (2001) included trust in their model of e-commerce
success. Trust in an e-commerce system has several benefits including heightened perceptions of
usefulness and greater intentions to use the system (Gefen et al., 2003). The latter argument may also be
extended to encompass the impact of trust on intentions to continue using an e-commerce system. Intention
to continue using a system is as much a behavioural intention as intention to use. Thus, the hypotheses
suggested are:
H5: Trust has a positive effect on Perceived Usefulness of an e-commerce system.
H6: Trust has a positive effect on Continuance Intention for an e-commerce system.
Molla & Licker (2001) provide support for the influence of trust on user satisfaction. Where an e-commerce
vendor is perceived as trustworthy, this leads to heightened levels of user satisfaction. The hypothesis
suggested is therefore that:
H7: Trust has a positive effect on User Satisfaction with an e-commerce system.
3.6 System quality

System quality in an e-commerce context, as with traditional IS, is reflected by usability, availability,
reliability, adaptability and fast response time of the system (DeLone & McLean, 2003). According to Seddon
(1997) system quality is concerned mainly with the consistency of the interface and the ease of use. As such
ease of use features prominently in the operationalisation of system quality (Ifinedo, 2006; Rai et al., 2002;
Seddon, 1997). In attempting to understand the relationship between system quality and other dimensions of
IS success the TAM can be referred to (Davis, 1989). In the TAM, perceived ease of use has been found to
influence both perceived usefulness and behavioural intentions to use a new system (Davis, 1989). Given
the prominence of ease of use in the assessment of system quality, it therefore follows that:
H8: System Quality has a positive effect on Perceived Usefulness of an e-commerce system.
H9: System Quality has a positive effect on Continuance Intention for an e-commerce system.
The positive impact of system quality on user satisfaction has been recognized and demonstrated in several
prior IS success studies (DeLone & McLean, 2003; Rai et al., 2002; Seddon & Kiew, 1996). Molla & Licker
www.ejise.com
170
(2001) argue that this effect applies equally to e-commerce systems. There is thus ample evidence to
support the hypothesis that:
H10: System Quality has a positive effect on User Satisfaction with an e-commerce system.
A link between system quality (ease of use specifically) and trust has been demonstrated by Gefen et al.
(2003). They posit that through investing time and effort in improving the ease of use and usability of
websites, e-commerce vendors demonstrate to consumers their integrity and trustworthiness. Thus, the
hypothesis supported is:
H11: System Quality has a positive effect on user Trust of an e-commerce system.
3.7 Information quality

Information quality is highlighted by DeLone & McLean (1992) as an important indicator of IS success.
Information quality has received increased attention since the advent of the Internet and World Wide Web
(WWW) (Lederer et al., 2000). Molla & Licker (2001) suggest that in the e-commerce context, it should be
referred to as content quality, whilst DeLone & McLean (2004) suggest retaining the label information quality.
Information and content quality are used interchangeably in this study. Lederer et al. (2000) found
information quality to be a major influence on the perceived usefulness of websites, as suggested by Seddon
(1997) and as validated by Rai et al. (2002) concerning traditional IS. Rai et al. (2002) also validated the
relationship between information quality and user satisfaction as suggested by DeLone & McLean (1992)
and Molla & Licker (2001). DeLone & McLean (2003) argue that information quality influences intentions to
use a system, which may equally apply to intentions to continue using a system. The hypotheses suggested
are therefore as follows:
H12: Information Quality has a positive effect on Perceived Usefulness of an e-commerce system.
H13: Information Quality has a positive effect on User Satisfaction with an e-commerce system.
H14: Information Quality has a positive effect on Continuance Intention with an e-commerce system.
3.8 Service quality

Pather et al. (2004) note the importance of service quality in the e-commerce context. Service quality (or
support and service in Molla & Licker, 2001) is defined as the overall support delivered by the e-commerce
service provider (DeLone & McLean, 2003). Molla & Licker (2001) postulate e-commerce satisfaction is
affected by the level of support and service (or service quality) provided by the e-vendor. DeLone & McLean
(2003) also show this relationship in their updated model of IS success. They further indicate that service
quality influences intentions to use a system, which may equally apply to intentions to continue using a
system. The hypotheses suggested are:
H15: Service Quality has a positive effect on User Satisfaction with an e-commerce system.
H16: Service Quality has a positive effect on Continuance Intention for an e-commerce system.
Harris & Goode (2004) found that service quality plays an important role, not only in enhancing user
satisfaction, but as an influence on trust and perceived usefulness. There is therefore support for the
following hypotheses:
H17: Service Quality has a positive effect on Trust in an e-commerce system.
H18: Service Quality has a positive effect on Perceived Usefulness of an e-commerce system.
The above set of hypotheses can be summarized into a research model, as shown in Figure 1 below.
www.ejise.com
171
ISSN 1566-6379
Figure 1: B2C E-commerce success model
4. Research methodology
The methodology employed in this study was positivistic, quantitative and hypothetic-deductive. Hypotheses
were derived from the extant literature on IS and e-commerce success, leading to the development of the
framework in Figure 1. A survey instrument was then developed in order to assess and measure the different
dimensions of success from an e-commerce user perspective. For the purposes of focusing the study, online
retail as a specific type of B2C e-commerce activity was investigated. The data was analysed statistically to
validate the instrument and test hypotheses. The instrument design, sample design, data collection
procedure and data analysis procedure will be discussed in this section.
4.1 Instrument design

DeLone & McLean (2004) provides the following advice to researchers aiming to evaluate e-commerce
success: Researchers and practitioners should not let themselves be carried away by the hype of the new
economy and led to believe that this new and rapidly changing environment requires entirely new measures
of IS success. One should look first at the cumulative tradition, and determine which existing and validated
success measures can be used in the e-commerce environment. As much as possible, tried and true
measures should be enhanced and expanded with modifications or, where necessary, new measures should
be considered. Selection of e-commerce success dimensions and measures should be contingent on the
objectives and the context of the empirical investigation, but tested and proven measures should be used
whenever possible. Completely new and untested metrics should be adopted only as a last resort (p. 43).
In keeping with this advice, measures that were tried and tested in previous IS success or e-commerce
success studies were sought. The measures needed to be sufficiently generalisable to various types of B2C
e-commerce systems. For example, online retail varies from online grocery shopping to airline ticket
purchasing, the frequency of purchase (and therefore e-commerce usage) varying depending on product and
context. A detailed search of relevant instruments used by previous researchers was conducted. Based on
this search, the instruments shown in Table 2 below were used to create the questionnaire. Slight
modification to suit the context of e-commerce was necessary for some instruments.
Table 2: Instruments Used to Measure B2C E-commerce Success Dimensions
Measure
System Quality
Information (Content) Quality
Service Quality (Support & Service)
Trust
Loyalty Incentives
User (Customer e-commerce) Satisfaction
Continuance Intention
www.ejise.com
Number of Items
7
7
4
6
3
6
1
3
172
Source
Seddon (1997)
Rai et al. (2002)
Wang et al. (2001)
Suh & Han (2002)
Bhattacherjee (2001)
Suh & Han (2002)
Rai et al. (2002)
Bhattacherjee (2001)
The questionnaire was divided into two sections. In the first section, the 8 measures in Table 2 were laid out
(See Appendix 1). The measures made use of a 5-point Likert scale, anchored by Strongly Disagree (1) at
one end to Strongly Agree (5) at the other. All the original measures in Table 2 were designed in this way.
The second section gathered general demographic information about the respondents such as age, income,
gender, Internet experience and habits, and e-commerce experience. Internet experience and habits were
assessed using measures derived from Tan & Teo (2000).
4.2 Sample design

This study was aimed at a population of e-commerce users, specifically online retail customers in South
Africa. There were no specific demographic requirements, apart from the fact that respondents were required
to have conducted online retail activity via the medium of the Internet. Of the estimated 3.4 million Internet
users in South Africa at the end of 2004 (World Wide Worx, 2005), about 41% were estimated to have
engaged in e-commerce activities such as shopping online (Webchek, 2004). The demographic profile of the
typical e-commerce user in South Africa is thought to be young to middle-aged, university-educated, and
affluent (De Villiers & Van Der Merwe, 2001; Brown & Buys, 2005). The gender split is estimated to be only
slightly biased towards males.
In order to reach a sample of this target population, postgraduate business students at a leading South
African university were the primary target group. A few respondents (7) from outside of this group also
participated. Postgraduate business students have been used in prior e-commerce studies in recognition that
they very often have e-commerce experience (Gefen at al., 2003). In South Africa this is the case too. Brown
& Buys (2005) found that this group conformed closely to the profile of the typical e-commerce user in terms
of age, education and income. A realistic sample for a positivistic study of this nature was deemed to be in
the region of 150 to 250 respondents (De Villiers & Van Der Merwe, 2001; Molla, 2002). This figure was
deemed more than sufficient to carry out the envisioned statistical analysis (Molla, 2002).
4.3 Pilot study

Before the major study, a pilot study was conducted. The purpose was to establish the basic soundness of
the instrument and to check for any problems related to wording and ambiguity in measurement items. The
respondents for the pilot study were 10 IS Masters students. Through the pilot it was recognized that
respondents needed to be instructed to base their responses on their most recent e-commerce experience,
rather than their general perceptions. Asking questions that relate to the most recent service encounter
enabled the researchers to elicit attitudes with respect to a specific experience (Shankar et al., 2003). The
instrument made use of existing validated measures, so no pre-tests were conducted to refine the measures.
4.4 Data collection and analysis procedure

For the postgraduate students, questionnaires were distributed just before the end of their formal classes.
This method was very effective as the questionnaires were returned immediately after completion. It also
ensured a high response rate. In total, 183 questionnaires were returned of which 166 were useable. 17
were unusable, either because they were incomplete or because the respondents indicated that they had no
experience of purchasing or shopping from online retail sites.
Several statistical tests were employed to analyse the data. Descriptive statistics were used to describe the
respondent profile, to ascertain the mean scores for key variables, and to determine correlations between
the variables. To test for instrument reliability, the Cronbach alpha was used, whilst factor analysis was
employed to determine construct validity. To test the hypotheses multiple linear regression equations were
formulated.
5. Data analysis and results

In this section, the demographic profile of the respondents and other descriptive findings will be presented,
followed by validation and refinement of the measuring instrument. The hypotheses will then be tested.
5.1 Demographic profile

Appendix 2 shows the self-reported demographic profile of the 166 valid responses in terms of gender, age,
education, profession and income. 69% of the respondents were male, with 93% between the ages of 25
44. 86% had either an undergraduate and/or postgraduate degree. 69% were managers or professionals,
www.ejise.com
173
ISSN 1566-6379
and 79% were in the upper income brackets, earning more than R 10,000 per month. This profile matches
the typical profile of a South African e-commerce user, except for the larger proportion of male respondents.
The male bias in the respondent profile was due to the majority of postgraduate business students being
male.
5.2 Internet and e-commerce experience

The level of Internet and e-commerce experience amongst the sample is shown in Appendix 3. 72% had
Internet access at home. The majority (71%) had over 5 years of Internet experience, but only 14% had over
5 years online shopping experience. Most (89%) used the Internet a few times a day or more, on average for
more than an hour at a time. The Internet was mostly used for communications, followed by getting
information and product support, downloading free resources, banking, online shopping and entertainment
respectively.
Respondents were asked what products and services they purchased online. The most common purchases
were by category: (1) books, magazines and stationary, followed by (2) DVDs, movies and videos, then (3)
music, and (4) flowers, gifts and cards. These results are consistent with those of De Villiers & Van Der
Merwe (2001), who also found that books were the most popular product purchased online by South African
consumers. The respondents were asked what online retail web site they were referring to in responding to
the questionnaire. The most commonly referred to was Amazon.com. Other retail web sites referred to
included Kalahari.net (a South Africa retailer similar to Amazon.com), FlySAA (a South African airline),
Picknpay.co.za (the e-commerce site of a major South African supermarket chain) and Inthebag.co.za (the ecommerce site of a major South African supermarket chain). This is also consistent with De Villiers & Van
Der Merwes (2001) findings where they stated: It should be of concern for local Internet entrepreneurs that
Amazon.com, an American site, is by far the most popular site for local Internet users (p. 13).
5.3 Instrument validity, reliability and refinement

The measures in the survey were derived from existing instruments that had been previously validated in
prior IS and e-commerce studies. It was still nevertheless necessary to confirm their validity and reliability.
Some refinements were also anticipated, since the context of e-commerce is slightly different to the more
traditional IS context where some of the measures had been developed.
5.3.1 Construct validity

Confirmatory factor analysis (CFA) is often used to assess construct validity (Molla, 2002). Since the
questionnaire was devised from previously validated instruments, CFA was used to validate and refine the
measurement model. The following commonly applied decision rules were used (Wang et al., 2001; Tan &
Teo, 2000):
Use a minimum eigenvalue of 1 as a cutoff value for extraction;

Delete items with factor loadings of less than 0.5 on all factors or greater than 0.5 on two or more
factors;
Use varimax rotation;
Exclude single item factors from the standpoint of parsimony.
There were 8 major constructs identified in the research model of Figure 1, containing 37 measurement
items, as shown in Table 2. One of the constructs, user (customer e-commerce) satisfaction contained only a
single item and so was not included in the factor analysis. 7 factors were therefore expected to emerge. The
iterative sequence of factor analysis and deletion was executed and repeated until distinct factors were
identified (Wang et al., 2001).
As a result of the CFA, 8 of the original 37 items mentioned in Table 2 were deleted to ensure construct
validity. The CFA identified 6 distinct factors, namely Perceived Usefulness/Continuance Intention,
Information (Content) Quality, System Quality, Trust, Loyalty Incentives and Service Quality (Support and
Service) (See Table 3). The only anomaly that remained was the loading of Continuance Intention on the
same factor as Perceived Usefulness. This attests to the strong influence of Perceived Usefulness on
Continuance Intention. These constructs were still retained and treated as separate constructs, as their
Variance Inflation Factors (VIFs) were well below the upper limit of 10 (Tan & Teo, 2000). Furthermore, there
are strong conceptual grounds for treating Perceived Usefulness and Behavioural Intentions as distinct
www.ejise.com
174
(Davis, 1989; Bhattacherjee, 2001). Table 3 shows the optimal factor loadings achieved after elimination of
items, while Table 4 illustrates which items were deleted, and the structure of the final measurement model.
Table 3: Final factor loadings
Perceived Usefulness (PU)
PU1
PU2
PU3
PU4
PU5
PU6
Information (Content) Quality (IQ)
IQ1
IQ2
IQ3
IQ5
IQ7
System Quality (SQ)
SQ4
SQ5 (Reversed)
SQ6 (Reversed)
SQ7 (Reversed)
Continuance Intention (CI)
CI1
CI3 (Reversed)
Trust (TR)
TR1
TR2
TR3
TR4
TR6
Loyalty Incentives (LI)
LI1
LI2
LI3 (Reversed)
Service (Support & Service) Quality (SS)
SS1
SS2
SS3
SS4
Expl.Var
Prp.Totl
Factor 1
Factor 2
Factor 3
Factor 4
Factor 5
Factor 6
0.10
-0.02
0.11
0.17
0.18
0.13
0.76
0.76
0.80
0.80
0.68
0.77
0.04
0.13
-0.08
-0.08
0.05
-0.10
0.06
0.14
0.13
0.00
0.11
0.28
0.16
0.08
0.17
0.05
-0.10
0.15
0.05
0.12
0.10
0.14
0.24
0.15
0.11
0.16
0.19
0.03
0.27
0.29
0.08
0.20
0.09
0.15
0.07
0.00
-0.01
-0.01
0.05
0.74
0.78
0.62
0.66
0.75
0.11
0.00
0.27
0.18
0.05
0.13
0.10
0.17
0.31
0.02
0.19
0.20
0.13
0.10
0.06
0.23
0.12
0.15
-0.01
0.05
0.02
0.09
0.21
0.10
-0.06
0.20
0.56
0.68
0.79
0.71
0.17
0.18
0.11
0.17
0.16
-0.10
0.56
0.55
0.02
-0.06
0.23
0.17
0.26
0.36
0.26
0.17
0.12
0.18
0.31
0.30
0.22
0.27
0.24
0.26
0.18
0.15
-0.04
0.01
-0.02
0.17
0.07
0.17
0.05
0.14
0.26
0.25
0.34
0.14
0.29
0.04
0.22
0.70
0.77
0.64
0.61
0.70
0.12
0.17
-0.04
0.02
-0.04
-0.03
0.90
0.92
0.81
0.02
0.03
0.02
0.10
0.08
-0.06
0.08
0.06
-0.02
0.73
0.73
0.76
0.76
2.95
0.10
0.13
0.19
0.12
0.13
4.69
0.16
0.05
0.12
0.12
0.05
2.46
0.08
0.14
0.15
0.27
0.24
3.21
0.11
0.33
0.29
0.06
0.12
2.81
0.10
0.20
0.23
0.25
0.22
3.04
0.10
Table 4: Refined Instruments

Construct
System Quality (SQ)
Service Quality (Support & Service) (SS)
Trust (TR)
User (Customer e-commerce) Satisfaction (US)
www.ejise.com
Original Instrument (No. of Items)

7
7
4
6
3
6
1
3
175
Refined Instrument (No. of Items)

4
5
4
5
3
6
1
2
ISSN 1566-6379
5.3.2 Instrument reliability

In order to assess reliability of the refined instrument, the Cronbach Alpha was calculated for each construct
(Tan & Teo, 2000). Reliability is indicated if the Cronbach Alpha is greater than 0.7. The lowest Cronbach
Alpha was 0.71, thus demonstrating that all measures exhibited reliability (see Table 5).
Table 5: Reliability test
Construct
Information Quality (IQ)
System Quality (SQ)
Trust (TR)
Service Quality (SS)
Items
6
5
4
5
3
4
2
Cronbach Alpha
0.89
0.83
0.76
0.86
0.86
0.88
0.71
User Satisfaction (US)
N/A
5.4 Descriptive statistics for B2C e-Commerce success dimensions

Table 6 illustrates the mean scores for the 8 dimensions and the correlations between them. The coefficients
in bold italic are significant at p < 0.05. The mean scores for all dimensions are above 3 on a scale of 1 to 5
indicating that on average the e-commerce systems were perceived as successful. The lowest score of 3
was for loyalty incentives (LI), and the highest was for continuance intentions (CI), with a mean of 4. All the
dimensions were significantly correlated with each other (p < 0.05), with the exception of loyalty incentives
(LI), which only correlated with service quality (SS). It is perhaps an indication that including LI in the ecommerce success model was ill-conceived. As DeLone & Mclean (2004) note, despite the
multidimensional and contingent nature of e-commerce success, an attempt should be made to significantly
reduce the number of different measures used to measure success, so that research results can be
compared and findings validated (p. 44). The 7 remaining factors are all interrelated, indicating they are part
of a higher order measure of B2C e-commerce success. The correlations between them do not exceed 0.63,
which is indicative that they are nevertheless distinct constructs.
Table 6: Descriptive Statistics for B2C e-Commerce Success Dimensions
Mean
PU
IQ
SQ
CI
TR
LI
SS
US
PU
IQ
SQ
CI
TR
LI
SS
3.7
3.4
3.8
4.0
3.7
3.0
3.4
1.00
0.41
1.00
0.37
0.38
1.00
0.63
0.41
0.46
1.00
0.50
0.50
0.55
0.50
1.00
0.01
0.08
0.11
0.00
0.12
1.00
0.37
0.51
0.49
0.36
0.62
0.20
1.00
0.44
0.40
0.45
0.48
0.53
0.13
0.55
US
3.9
1.00
5.5 Hypothesis testing

The hypotheses were tested using multiple linear regression equations to represent the relationships shown
in Figure 1. The following equations were built:
CI = h1* US + h3 * PU + h4 * LI + h7 * TR + h9 * SQ + h14 * IQ + h16 * SS.(i)
US = h2 * PU + h6 * TR + h10 * SQ + h13 * IQ + h15 * SS.....(ii)
PU = h5 * TR + h8 * SQ + h12 * IQ + h18 * SS....(iii)
TR = h11 * SQ + h17 * SS..(iv)
h1 to h18 represent the beta values corresponding to the hypotheses they represent. Recursive
simultaneous equation models, such as equations (i) to (iv) can be estimated by using multiple linear
regression applied to each equation separately (Pindyck & Rubinfeld, 1998). Table 7 below summarises the
results of the regression tests. Hypotheses were supported if the p values were less than 0.05. 9 of the 18
hypotheses were supported and are highlighted in bold italics in Table 7. Table 7 shows that the three major
variables influencing continuance intentions are perceived usefulness, user satisfaction and system quality,
www.ejise.com
176
with perceived usefulness being the primary contributor. User satisfaction in turn is influenced by perceived
usefulness and service quality, with service quality having the higher beta value. Perceived usefulness is
primarily influenced by trust and information quality. Trust is strongly influenced by service quality and
system quality, in that order.
Table 7: Hypothesis testing
Hypothesis
Independent Variable
Dependent Variable
Beta
value
p level
(*p<0.05)
Hypothesis
Supported?
H1
User Satisfaction
(US)
(CI)
0.17
*0.0223
Yes
H2
Perceived
Usefulness (PU)
User Satisfaction
(US)
0.18
*0.0131
Yes
H3
Perceived
Usefulness (PU)
(CI)
0.43
*0.000
Yes
H4
Loyalty Incentives
(LI)
(CI)
-0.05
0.431
No
H5
Trust (TR)
(PU)
0.34
*0.0004
Yes
H6
Trust (TR)
0.16
0.0676
No
H7
Trust (TR)
0.11
0.1900
No
H8
System Quality (SQ)
0.11
0.1848
No
H9
System Quality (SQ)
0.17
*0.0150
Yes
H10
System Quality (SQ)
0.14
0.0752
No
H11
System Quality (SQ)
Trust (TR)
0.31
0.0000
Yes
Information Quality
(IQ)
Information Quality
(IQ)
(PU)
User Satisfaction
(US)
0.19
0.0175
Yes
0.05
0.5112
No
H14
Information Quality
(IQ)
(CI)
0.09
0.2114
No
H15
User Satisfaction
(US)
0.28
0.0008
Yes
H16
(CI)
-0.08.
0.3115
No
H17
Trust (TR)
0.47
0.0000
Yes
H18
(PU)
0.01
0.9215
No
H12
H13
User Satisfaction
(US)
(CI)
(PU)
(CI)
User Satisfaction
(US)
The supported hypotheses are illustrated in Figure 2 below.
www.ejise.com
177
ISSN 1566-6379
Figure 2: Refined e-Commerce success model
6. Discussion and implications

The model shown in Figure 2 is a very much refined version of Figure 1. As illustrated in Table 6 all the 7
dimensions in Figure 2 are significantly correlated, indicating that they are all dimensions of the same higher
order e-commerce success construct. The contribution of multiple linear regression analysis has been to
identify the most significant relationships while controlling for interrelationships between factors and indirect
effects. The strong influence of perceived usefulness and user satisfaction on continuance intention and the
negligible influence of loyalty incentives are as expected from Bhattacherjee (2001). The influence of system
quality on continuance intentions mirrors the equivalent perceived ease-of-use to usage-intentions
relationship in the pre-adoption TAM model (Davis, 1989). System quality is often operationalised as the
ease and/or difficulty of system use (Rai et al., 2002; Seddon, 1997), which allows for this parallel to be
drawn. The influence of information quality is shown to be primarily on perceived usefulness and not
continuance intentions. This finding is indicative that information quality indirectly influences continuance
intentions through perceived usefulness, as reasoned by Lederer et al. (2000). This rationale may apply also
to the trust variable which has a strong influence on perceived usefulness, but not on continuance intentions.
Perceived usefulness and service quality both have an influence on user satisfaction. Trust indirectly affects
user satisfaction through perceived usefulness. Trust, finally, is strongly influenced by service and system
quality. The relationship between system quality and perceived usefulness is surprisingly absent in the
refined model of Figure 2. This apparent anomaly can be attributed to the inclusion of the trust variable,
whereby system quality indirectly affects perceived usefulness through trust.
These findings show that the inclusion of service quality (DeLone & McLean, 2003) and trust (Molla & Licker,
2001) in the traditional IS success model significantly alters the nature of relationships between success
dimensions. These relationships are reflective of the new e-commerce context, where service quality and
trust are essential attributes for successful B2C e-commerce. The inclusion of continuance intentions rather
than use or intentions to use is also a valuable refinement, as customer intentions to continue using an ecommerce system are reflective of repeat customers. Given the importance of these three variables to ecommerce success, more sophisticated conceptualizations and measures can be used to investigate their
full impact. For example, Gefen et al. (2003) identify structural assurances, familiarity with the e-vendor,
situational normality and calculative-based beliefs as important factors in enhancing trust. Concerning
service quality, several studies suggest the use of a comprehensive instrument such as SERVQUAL to
assess this dimension (DeLone & McLean, 2004; Molla & Licker, 2001; Pather et al., 2004).
DeLone & Mclean (2004) recommend that fewer and not more dimensions are needed to describe IS
success. The inclusion of continuance intention was not as an extension to their model, but as a better
means of assessing usage intentions in a post-adoption context such as when evaluating IS success
(Bhattacherjee, 2001). The inclusion of trust as an additional variable was justifiable given its importance to
e-commerce (Gefen et al., 2003; Molla & Licker, 2001). The addition of loyalty incentives on the other hand
was found to be unnecessary. This additional factor did not correlate with most of the other interrelated
dimensions. It also did not influence continuance intentions as hypothesized. DeLone & Mcleans (2004)
www.ejise.com
178
stance that additional dimensions beyond those in their updated model will not add much value to the ecommerce success concept is therefore partially vindicated.
In developing an instrument to assess the different dimensions of success, existing validated measures were
drawn upon and modified appropriately. The use of existing measures proved successful as the instrument,
after refinement, was shown to be both valid and reliable. DeLone & McLeans (2004) exhortation to
researchers to use existing measures so as to build up a cumulative tradition was therefore justified.
The model of Figure 2 provides practitioners with a holistic, multidimensional perspective on e-commerce
success. There is a tendency in practice to focus on different success dimensions in isolation, and not
recognize their strong interrelationships. By considering the different dimensions in the model, and looking
closely at their interrelationships, a better understanding of how to measure and improve e-commerce
offerings can be obtained. The model may also be used to diagnose and address problems with B2C ecommerce offerings. For example, if an online vendor is experiencing problems related to lack of trust
amongst potential or existing e-customers, a two-pronged strategy aimed at simultaneously enhancing
service quality and system quality will help improve the levels of trust.
7. Limitations and further research

The sample was drawn largely from a group of MBA and other postgraduate students at a leading South
African university. Most of the respondents were working professionals or managers and were representative
of the typical e-commerce user in South Africa. There was however a predominance of males in the sample,
whereas there is no major gender bias evident amongst e-commerce users in South Africa (De Villiers & Van
Der Merwe, 2001). This was a limitation of using such a group of students. Future research might draw from
a wider sample of e-commerce users to ensure the sample is even more representative of the typical ecommerce user in South Africa.
The instrument validation process revealed that continuance intention and perceived usefulness loaded on
the same factor during CFA. This was indicative of the strong relationship between perceived usefulness and
behavioural intentions, as identified in the pre-adoption TAM model, and in post-adoption studies such as by
Bhattacherjee (2001). To ensure greater discriminant validity, it may be necessary to revisit the continuance
intention instrument, given that the perceived usefulness instrument has been validated in a wide variety of
contexts.
For the sake of parsimony, it may also be suggested that either perceived usefulness or continuance
intentions be included in the e-commerce success model. This would be in line with the call by DeLone &
McLean (2004) for a reduction in the number of dimensions of e-commerce success. Future research could
therefore assess alternative models of e-commerce success to identify the best-fit model. Rai et al. (2002)
conducted such a study for traditional IS success models.
Some dimensions of e-commerce success were difficult to operationalise. For example, user satisfaction with
e-commerce has been identified by Wang et al. (2001) as consisting of 7 dimensions. Many of the
dimensions overlap with those identified in the e-commerce success model of Figure 1. A single-item
measure was therefore used to assess overall user satisfaction as suggested by Rai et al. (2002). Future
research ought to bring conceptual clarity to this overlap of dimensions between e-commerce user
satisfaction and success.
This study has incorporated continuance intentions as the ultimate dependent variable. Future research
might attempt to incorporate appropriate measures for use and net benefits into the model, either as an
alternative for continuance intentions or in addition to it.
To add explanatory richness and greater conceptual clarity to the phenomenon of B2C e-commerce
success, it is perhaps timely at this juncture to conduct an inductive, qualitative enquiry using methods such
as the grounded theory methodology. Even in well-researched areas such as IS success, grounded theory
studies may be conducted to garner a better understanding of the social processes at play (Glaser, 1992).
8. Conclusion
IS success is a multi-faceted and multi-dimensional construct. In a B2C e-commerce context, customer
service and trust in an online vendor are of utmost importance. It stands to reason therefore that service
www.ejise.com
179
ISSN 1566-6379
quality and trust are two additional dimensions to consider in addition to the traditional dimensions of system
quality, information quality, user satisfaction, perceived usefulness, and use/intentions to use.
Given the variety of products and services on offer to e-commerce consumers, using frequency of use as a
general measure of e-commerce success is problematic. Frequency of use is also dependent on the nature
of products and services being purchased. For example, online grocery shopping might be conducted more
frequently than purchase of airline tickets for some. Assessing customer intentions to continue using an ecommerce system is perhaps a better measure of success, as it reflects repeat business regardless of
product/service. It also treats behavioural intentions as a post-adoption phenomenon, as is required for
evaluating success of a system. The pre-adoption intention to use measure from the TAM model is not as
suitable to use for this evaluative purpose.
The validation of the initially formulated model in Figure 1 showed that the 7 major dimensions making up ecommerce success are all interrelated. However, only 9 of the 18 original hypotheses were supported. This
was because the hypothesis testing delineated direct and indirect relationships between the core set of ecommerce dimensions. Overall the ultimate dependent variable, continuance intentions, was found to be
directly influenced by perceived usefulness, user satisfaction and system quality. User satisfaction was
directly impacted by service quality first and foremost and then perceived usefulness secondly. This affirms
the strong association between a satisfied customer and good quality service. Perceived usefulness was
influenced by trust and information quality. Trust in turn was influenced by service and system quality.
The confirmation of 7 interrelated dimensions for e-commerce success and a set of 9 direct relationships
provides for a parsimonious way of understanding and evaluating B2C e-commerce success. Much of the
research in this domain has been of a conceptual nature. This study adds to the body of knowledge by
empirically validating a refined model of B2C e-commerce success upon which future studies can build.
References
Bhattacherjee, A (2001). An Empirical Analysis of the Antecedents of Electronic Commerce Service Continuance,
Decisions Support Systems, Vol. 32, Iss. 2, pp 201214.
Brown, I and Buys, M (2005). Customer Satisfaction with Internet Banking Web Sites:
An Empirical Test and Validation of a Measuring Instrument, South African Computer Journal, Vol. 39, pp 29 37.
Cyr, D, Bonanni, C and Ilsever, J (2004). Design and e-Loyalty across Cultures in Electronic Commerce, Proceedings
of the 6th International Conference on Electronic Commerce, 2004. Delft: Netherlands, pp 351-360.
Davis, F (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS
Quarterly, Vol. 13, Iss. 3, pp 319-339.
DeLone, W and McLean, E (1992). The Quest for the Dependent Variable, Information Systems Research, Vol. 3, Iss.
1, pp 60-65.
DeLone, W and McLean, E (2003). The DeLone and McLean Model of Information Systems Success A 10 Year
Update, Journal of Management Information Systems, Vol. 19, Iss. 4, pp 9-30.
DeLone, W and McLean, E (2004). Measuring e-Commerce Success: Applying the DeLone & McLean Information
Systems Success Model, International Journal of Electronic Commerce, Vol. 9, Iss. 1, pp 31 47.
De Villiers, C and Van Der Merwe, J (2001). An Investigation into the Adoption of Electronic Commerce by South African
Consumers Who They Are and What They Think. Technical Report, University of Pretoria.
Fusilier, M and Durlabhji, S (2003). No Downturn Here: Tracking e-Business Programs in Higher Education, Decision
Sciences Journal of Innovative Education, Vol. 1, Iss. 1, pp 73-98.
Garrity, E, Glassberg, B, Kim, Y, Sanders, G and Shin, S (2005). An Experimental Investigation of Web-based
Information Systems Success in the Context of Electronic Commerce, Decision Support Systems, Vol. 39, pp 485503.
Gefen, D, Karahanna, E and Straub, D (2003). Trust and TAM in Online Shopping: An Integrated Model, MIS Quarterly,
Vol. 27, Iss. 1, pp 51-90.
Glaser, B (1992). Emergence vs. Forcing. Basics of Grounded Theory Analysis. Sociology Press, Mill Valley.
Goldstuck, A (2002). The Goldstuck Report: Online Retail in South Africa, 2002. Pinegowrie: World Wide Worx.
Goldstuck, A (2004). Online Retail in South Africa 2004. Pinegowrie: World Wide Worx.
Grandon, E and Pearson, J (2004). Electronic Commerce Adoption: An Empirical Study of Small and Medium US
Businesses, Information & Management, Vol. 42, pp 197-216.
Harris, L and Goode, M (2004). The Four Levels of Loyalty and the Pivotal Role of Trust: A Study of Online Service
Dynamics, Journal of Retailing, Vol. 80, pp 139158.
Ifinedo, P (2006). Extending the Gable et al. Enterprise Systems Success Measurement Model: A Preliminary Study,
Journal of Information Technology Management, Vol. 17, Iss. 1, pp 14 33.
Iivari, J (2005). An Empirical Test of the DeLone-McLean Model of Information System Success, DATA BASE for
Advances in Information Systems, Vol. 36, Iss. 2, pp 8-27.
Kim, M, Kim, W and Oh, S (2006). Past, Present and Future of e-Business, International Journal of Contents, Vol. 2,
Iss. 1, pp 1-4.
www.ejise.com
180
Lederer, A, Maupin, D, Sena, M and Zhuang, Y (2000). The Technology Acceptance Model and the World Wide Web,
Decision Support Systems, Vol. 29, Iss. 3, pp 269282.
Molla, A. (2002). E-Readiness and E-Commerce Success: Developing and Exploring an Antecedent Model in Developing
Countries Context. Unpublished PhD Thesis, University of Cape Town.
Molla, A and Licker, P (2001). E-Commerce Systems Success: An Attempt to Extend and Respecify the Delone and
Mclean Model of Information Systems Success, Journal of Electronic Commerce Research, Vol. 2, Iss. 4, pp 1-11.
Parker, C and Castleman, T (2007). New Directions for Research on SME e-Business: Insights from an Analysis of
Journal Articles from 2003 to 2006, Journal of Information Systems and Small Business, Vol. 1, Iss. 1-2, pp 21-40.
Pather, S, Remenyi, D and Erwin, G (2004). E-Commerce Success: The Quest for IS Effectiveness Measurement: A
Conceptual Framework for the e-Commerce Environment, South African Computer Journal, Vol. 32, pp 34 43.
Pather S, Remenyi D and de la Harpe A (2006). Evaluating e-Commerce Success A Case Study The Electronic
Journal of Information Systems Evaluation, Vol. 9, Iss. 1, pp 15-26, available online at www.ejise.com.
Pavic, S, Koh, S, Simpson, M and Padmore, J (2007). Could e-Business Create a Competitive Advantage in UK
SMEs? Benchmarking: An International Journal, Vol. 14, Iss. 3, pp 320-351.
Pavlou, P and Fygenson, M (2006). Understanding and Predicting Electronic Commerce Adoption: An Extension of the
Theory of Planned Behavior, MIS Quarterly, Vol 30, Iss 1, pp 115-143.
Pindyck, R and Rubinfeld, D (1998). Econometrics Model and Economic Forecasts (4th Edition), McGraw-Hill,
Singapore.
Quaddus, M and Achjari, D (2005). A Model for Electronic Commerce Success, Telecommunications Policy, Vol. 29, pp
127-152.
Rai, A, Lang, S, and Welker, R (2002). Assessing the Validity of IS Success Models: An Empirical Test and Theoretical
Analysis, Information Systems Research, Vol. 13, Iss. 1, pp 50-69.
Seddon, P (1997). A Respecification and Extension of the DeLone and Mclean Model of IS Success, Information
Systems Research, Vol. 8, Iss. 3, pp 240-253.
Seddon, P and Kiew, M (1996). A Partial Test and Development of DeLone and Mcleans Model of IS Success,
Australian Journal of Information Systems, Vol. 4, Iss. 3, pp 66-88.
Shankar, V, Smith, A and Rangaswamy, A (2003). Customer Satisfaction and Loyalty in Online and Offline
Environments, International Journal of Research in Marketing, Vol. 20, Iss. 2, pp 153175.
Suh, B and Han, I (2002). Effect of Trust on Customer Acceptance of Internet Banking, Electronic Commerce Research
and Applications, Vol. 1, pp 247263.
Tan, M and Teo, T (2000). Factors Influencing the Adoption of Internet Banking, Journal for the Association of
Information Systems, Vol. 1, Iss. 5, pp 1-42.
Torkzadeh, G and Dhillon, G (2002). Measuring Factors that Influence the Success of Internet Commerce, Information
Systems Research, Vol. 13, Iss. 2, pp 187-204.
Van Slyke, C, Belanger, F and Comunale, C (2004). Factors Influencing the Adoption of Web-Based Shopping: The
Impact of Trust. DATA BASE for Advances in Information Systems, Vol. 35, Iss. 2, pp 32-49.
Wang, T, Tang, T and Tang, J (2001). An Instrument for Measuring Customer Satisfaction toward Web Sites that Market
Digital Products and Services, Journal of Electronic Commerce Research, Vol. 2, Iss. 3, pp 20-48.
Webchek (2004). Online Shopping Trends October 2004, available online at
http://www.webchek.co.za/library_trends2.html. (Accessed 18 July 2006).
World Wide Worx (2005). Great Expectations Stall on the Internet, available online at
http://www.theworx.biz/access05.htm (Accessed 18 July 2006).
World Wide Worx (2006). Online Retail in South Africa 2006 Broadband and Experience the Keys to
Online Retail, available online at http://www.theworx.biz/retail06.htm (Accessed 18 July 2006).
Appendix 1: Instrument Measures

(Answered with respect to the most recent e-commerce experience)

PU1. Using this online shopping site enhances the productivity of my shopping activities.
PU2. Using this online shopping site has a critical role in supporting my shopping activities.
PU3. Using this online shopping site makes it easier to do my shopping activities.
PU4. Using this online shopping site enables me to accomplish shopping activities more quickly.
PU5. Using this online shopping site improves my performance (e.g. saving time or money) of shopping
activities.
PU6. I find this online shopping site useful for my shopping activities.

IQ1. This online retail site provides the precise information I need.
IQ2. This online retail site provides responses to questions and queries that are exactly what I need.
IQ3. This online retail site provides sufficient information to enable me to do my tasks.
* IQ4. This online retail site has errors that I must work around.
www.ejise.com
181
ISSN 1566-6379
IQ5. I am satisfied with the accuracy of information provided by this online retail site.
* IQ6. The output options (printer-friendly options, page sizes allowed for, etc.) are sufficient for my use.
IQ7 .The information provided by this online retail site is helpful regarding my questions or problems.
System quality (SQ)

* SQ1. This online retail site is user friendly.
* SQ2. Compared to other sites, this online retail site is easy to become familiarized with.
* SQ3. I find it easy to get this online retail site to do what I want it to do.
SQ4. I find it easy to become skilful at using this online retail site.
SQ5. I believe that this online retail site is cumbersome to use.
SQ6. To use this online retail site requires a lot of mental effort.
SQ7. I get frustrated when I use this online retail site.

CI1. I want to continue using this online retail site rather than discontinue its use.
* CI2. My intentions are to continue using this online retail site rather than use any other alternative means of
shopping.
CI3. I would like to discontinue the use of this online retail site.
Trust (TR)
TR1. This online retail site is trustworthy.
TR2. I trust in the benefits of the decisions made by this online retail site, i.e.: I trust the specials offered and
recommendations made by the site.
TR3. This online retail site keeps its promises and commitments.
TR4. This online retail site keeps customers best interests in mind.
* TR5. This online retail site would do the job right even if not monitored.
TR6. I trust this online retail site.
Loyalty incentives (LI)

LI1. This online retail site offers incentives for its continued use, such as frequent flier miles or bonus points.
LI2. I get rewarded for my continued patronage of this online retail site.
LI3. This online retail site generally does not give me any loyalty incentives for my continued use of its
service.
Service quality (support and service) (SS)

SS1. I am satisfied with the customer support provided by this online retail site.
SS2. I am satisfied with the after-sales service provided by this online retail site.
SS3. This online retail site understands my problems and requests.
SS4. This online retail site responds to my requests fast enough.
User (customer e-Commerce) satisfaction (US)

US1. I am satisfied with making purchases online from this online retail site.
* Dropped Items
www.ejise.com
182
Appendix 2: Demographic profile of respondents

Gender
Male
Female
Unknown
Count
114
51
1
69%
31%
1%
Age
Under 18
3
18 24
4
25 34
113
35 44
42
45 -54
4
55 +
0
Highest Education Qualification
High School
4
College Diploma
9
Technikon Diploma
11
Undergrad. Degree
49
Postgrad. Degree
93
Occupation
Full-time Student
36
Professional
76
Technician/Artisan
4
Manager
38
Unemployed
1
Self-employed
4
Other
5
Missing
2
Monthly Taxable Income
R 0 - R 1,999
11
R 2,000 - R 4,999
5
R 5,000 - R 9,999
11
R 10,000 - R 19,999
54
R 20,000 +
76
Missing
www.ejise.com
183
2%
2%
68%
25%
2%
0%
2%
5%
7%
30%
56%
22%
46%
2%
23%
1%
2%
3%
1%
7%
3%
7%
33%
46%
5%
ISSN 1566-6379
Appendix 3: Internet and e-Commerce experience

Internet Access at Home?
Yes
No
Missing
Internet Experience (yrs)
<1
13
34
45
>5
Online Shopping Experience (yrs)
<1
13
34
45
>5
Count
119
46
1
72%
28%
1%
2
9
15
22
118
1%
5%
9%
13%
71%
31
64
33
15
23
19%
39%
20%
9%
14%
Frequency of Internet Use

Few times/month
8
5%
Once a week
10
6%
Few times/day or more
148
89%
Intensity of Internet Use per Average Day (hrs)
<
12
7%
1/2 1
49
30%
13
68
41%
>3
37
22%
Extent of Internet Use (Scale of 1 to 5)
Mean
StdDev
Get Information and Product Support 4.0
1.1
Communications
4.8
0.8
Download Free Resources
3.3
1.3
Entertainment (e.g. Online games)
1.9
1.2
Purchasing/Shopping
2.6
0.8
Banking
www.ejise.com
3.2
184
1.1
Important Issues for Evaluating Inter-Organizational Data

Integration Configurations
Frank G. Goethals
IESEG School of Management, Lille, France
F.Goethals@ieseg.fr
Abstract: Partnering companies can share data via various configurations. Typically problems become evident as
partnering companies seek to exchange data. These problems take a different form for different data integration
configurations and are thus of great relevance when evaluating different configurations. This paper identifies issues to be
taken into account when evaluating inter-organizational data integration configurations. Eight problems are discussed:
difficulties to identify which information flows to automate; to relate different viewpoints on boundary objects; to agree on
data formats; to distribute investments among the parties; to deliver appropriate service levels; to preserve the value of
the data sharing; to clarify data ownership and to do all of this in the frame of changing relationships. For several
problems it is illustrated how they surface in a completely centralized and in a completely decentralized interorganizational data-integration scenario.
Keywords: business-to-business integration, inter-organizational data integration problems, boundary objects, service
levels, data ownership, data functionality
1. Introduction
Optimization efforts often focus on inter-organizational activities. Such schemes heavily rely on information
sharing for two principal reasons: (1) as information is being shared, data inconsistencies across enterprises
can be eliminated so that all companies accurately perceive the current state and (2) as new information is
being shared, new business practices become possible.
The process-paradigm indicates companies can execute a task if they are in a specific state. Information and
Communication Technology (ICT) makes it possible to transmit information about a multitude of states in
real-time. For example, in the past only two states were recognized in the ordering process: order placed
and delivery received. Nowadays a customer company can be informed that an order was received,
accepted, scheduled, picked, loaded in the truck, etc.
A number of information sharing problems are associated with setting up an inter-organizational system to
share data. Section 2 identifies eight problems and illustrates them in the context of the following practices:
Vendor Managed Inventory (VMI). The Bullwhip Effect is a significant problem that is discussed
extensively in Supply Chain Management literature. Variability in demand is magnified at each stage up
the supply chain (i.e., from reseller, over seller to manufacturer). Case studies have proven that, through
Vendor Managed Inventory, the Bullwhip-effect can be strongly reduced (Smaros et al. 2003). VMI
requires an intensive sharing of stock and sales data among different companies in the supply chain.
Information exchange in this configuration is so extensive that it is unrealistic to assume the data could
be shared and processed manually, without directly connecting the computer systems of the different
companies.
Product Lifecycle Management (PLM). According to CIMdata (2005), PLM involves the collaborative
creation, management, dissemination and use of product definition information across the Extended
Enterprise from concept to end of life of the product. Takalikar (2004) asserts companies with PLM get
better products to market faster and provide better support to the customers.
- A specific case: e-procurement via Tradcom. Tradcom (Muylle & Croon 2003) offers a virtual trading
environment for companies in Belgium, the Netherlands and Luxembourg to trade indirect goods and
services. Neef (2001, p. 25) defines such indirect goods as any commodity or service that a company
buys that does not result directly in finished goods. Tradcoms suppliers tend to enjoy a long term
relationship with Tradcom. Many of the suppliers now co-own Tradcom. Computer systems between
suppliers and Tradcom are highly integrated. Customer data couplings are looser. Since this trade
concerns indirect goods, customers typically do not want to invest heavily in linking their systems directly
to the platform.
After presenting eight problems associated with sharing data in Section 2, Section 3 illustrates how the
problems take a different form in different data sharing configurations. This leads to the conclusion that those
problems have to be considered when evaluating inter-organizational data integration configurations. The
ISSN 1566-6379
185
Goethals, F. G. Important Issues for Evaluating Inter-Organizational Data Integration Configurations. The Electronic
Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 185 - 196, available online at www.ejise.com
goal of the paper is not to give an exhaustive list of issues that have to be considered. Rather, eight
problems are discussed which were found to be of relevance in several cases.
It should be noted that this paper investigates only the exchange of explicit knowledge. Tacit knowledge is at
least until some point in time not stored in computer systems (see Nonaka 1994). As such, there are a
number of distinct characteristics which make it inappropriate to treat explicit and tacit knowledge as one and
the same.
2. Problems associated with sharing data

Case studies (such as the Tradcom case study) and literature (Ljung 2004; Sheu et al. 2001; Tang et al.
2006; etcetera) indicate that a number of important problems become evident when companies begin to
share information. This section identifies eight frequently encountered problems. Each problem is defined
and its relevance is illustrated by referring to one or more of the practices presented above: VMI, PLM and eprocurement via Tradcom. This paper focuses on identifying principal problems. Further research is required
to elaborate rectification projects.
The eight problems we discuss are the following:
1. First and foremost, companies have to define what information flows are valuable from a business
viewpoint: what data does a company want to use, when should it get that data for the data to be useful, etc.
Identifying what information flows to automate is not an easy task.
2. When one wants to identify information flows, one will stumble across another problem: if companies
want to share data about an object, they have to acknowledge they may use the object in different tasks and
they, therefore, may have another viewpoint on the object. Thus they need to map their viewpoints before
they can actually go about sharing data.
3. At this stage partners know what data they intend to share. The next phase requires defining the data
format.
4. To realize the information sharing, investments will need to be made. Partners have to agree on who will
bear costs for installing, maintaining and upgrading the systems.
5. Once the investments are made, business continuity can only be ensured if the data is provided by the
systems as needed. The systems should thus offer appropriate service levels in terms of availability,
response time, etc.
6. Sharing data gives the parties involved new sources of power. Data receivers may forward the data to a
third party or may inadequately secure their systems, the data provider may not pay enough attention to data
quality, etc. Partners must preserve the value of the data sharing.
7. A party should be designated that can decide what can or has to happen with specific data, and what is
prohibited. This requires clarifying who the designated data owner is.
8. All of the problems described above have to be dealt with in the frame of changing relationships. New
partners may be added, and partners may be removed from the network.
2.1 Valuable information sharing practices have to be identified

Partners have to identify what information sharing practices add value.
2.1.1 Problem description

Identifying the content, timing and parties involved for an electronic data transmission is far from
straightforward. When starting a Business-to-Business integration (B2Bi) effort it is very likely that the
existing information flows between the partners have not been architected or made explicit (Goethals et al.
2006b). This means many existing information flows are invisible, complicating their digitalization. Moreover,
creativity and coordination are needed to redesign existing information flows and identify valuable new
information flows and ways to realize them. A party can be required to capture new or more fine-grained data
internally. New data content can be identified that only exists at the level of the Extended Enterprise.
2.1.2 Illustration of relevance

The introduction of Electronic Data Interchange (EDI) enabled companies to digitally transmit existing
documents such as purchase orders. This way, the traditional replenishment process was automated. Later
on, creative minds introduced VMI as a substitute for the traditional replenishment process, relying upon the
www.ejise.com
186
Frank G. Goethals
introduction of new information flows that were also based on EDI. Sellers only used to receive (and to have
knowledge about the resellers purchase orders. With VMI, sellers regularly receive information about the
resellers sales orders and stock states (Angulo et al. 2004; Danese 2004). Pursuing VMI requires
agreement on what messages to transmit and the timing of message transmissions. Messages may only be
transmitted after a request or regularly as a subscription with a publication once a week, daily, several times
per day, etcetera.
PLM acknowledges regular transmissions of information between different designers are valuable. Sharing
information among different designers requires many decisions to be made, the timing of transmission for
example. If one is reworking an artifact that is being used by others in their decision making process, should
a new version of the design then be transmitted to the others every 15 minutes, every hour, every day or
every week?
2.2 Partners have different viewpoints on objects

The determination of what data to share is complicated by the fact that the partners have a different
perception of the objects on which they want to share data.

Successful data exchanges between two parties generally fit with theories on boundary objects (or trading
zones) (Star & Griesemer 1989; Chrisman 1999). Boundary objects are objects that both inhabit several
intersecting social worlds and satisfy the informational requirements of each of them. Boundary objects are
objects which are both plastic enough to adapt to local needs and the constraints of the several parties
employing them, yet robust enough to maintain a common identity across sites (Star & Griesemer 1989, p
393). A flight is an example of a boundary object:
A pilot flying an airplane needs to know where a flight is going to, but not who is sitting where on the
plane.
The stewards have to know where everyone is to be seated on the plane.
The catering company needs to make sure that the right type and quantity of food is available for a
specific flight. Appropriate catering is a function of
the number of people seated on the plane, what is related to the identity of the people on the plane,
known by the stewards; and possibly of
the location where the plane is heading to/from, known by the pilots.
Carlile (2002) asserted that knowledge is localized and embedded. Integrating the data about an object is
problematic as different user groups have different perceptions of the object, having common and diverse
needs and interests. They execute different tasks using the object data and different business rules
pertaining to the object are relevant. User groups views are interrelated and information in different views
needs to be aligned.
The ultimate level of transparency is to integrate enterprises so that only one electronic version of an order
is saved in a shared database and that Purchase Order and Sales Order melt entirely together. Making this
vision reality is difficult because companies involved interpret the concept of order differently. That is, an
order is a boundary object. Simply stated, different companies can have a number of common and specific
events that may affect the state of the object. For common events, it has generally been acknowledged that
the lifecycles of the different perceptions of the boundary object should be compatible. Snoeck et al. (2004)
for example show that both companies should agree whether an order first has to be paid for and then has to
be delivered; or the other way around. This does not create an additional burden on the goal to store only
one copy of the boundary object. However, when it comes to the specific events, the boundary object is
brought in a different state for one company, but not in another state for the other company. For example,
checking the customer history can categorize the order into the state urgent handling or the state
postponed handling for the supplier, while the customer leaves it in the state PO sent. Consequently, not
all information on the boundary object simply can be stored in a single shared electronic object.

Carlile (2002) studied four principal functions that are dependent upon each other in the creation and
production of a product: (1) sales/marketing, (2) design engineering, (3) manufacturing engineering and (4)
www.ejise.com
187
ISSN 1566-6379
production. He found that knowledge specialization makes it complex to work across functional boundaries
and to accommodate the knowledge created in another practice.
People perceive product data differently. Designers make a Computer Aided Design (CAD) file and machine
operators focus on Numerical Control (NC) files. Workmen that install a part are only interested in the outside
appearance of the part and whether that part fits. A designer considers the same object as vector graphics.
Consumers of a product want to see how the product appears in different settings (e.g., a refrigerator placed
in different kitchens). Engineering Bill of Materials (BOM) can be changed to create a manufacturing BOM by
adding information about machines that will be used during production. Although all data concerns the same
product, only to some extent are the various parties interested in the same information.
To determine the relationship between data one has and data others need, the events that happen to an
object have to be investigated. Several parties influence the state of the object and the data that is valid is
thus determined by many parties together.
All partners need data that is valid to them. This validity does not require consistency, as events happen to
the object that are not relevant to a party. Data that is valid for one company is not necessarily valid for
another company. For instance, an engineering company decides to forward its entire engineering BOM to a
number of suppliers, each of which is only interested in a subpart of the BOM. Updates to the BOM are
relevant for one party, but not for another. That is, without the update the data is not consistent, but still valid
for some suppliers.
2.3 An appropriate data format has to be defined

Once the information companies want to share is known, partners have to look for a way to share data so
that it offers desired functionality.

Choosing a data format is not easy. Partners can have data in systems they have developed themselves or
that were created by vendors. These systems are likely to support different data formats and their interfaces
may not be documented. Even though many XML industry standards have been developed to exchange
data, many companies have legacy investments in EDI systems. These make the choice for XML messages
less evident. An important decision companies have to make concerns how they annotate and structure
data. Different formats can be adopted to transmit data. Different formats, however, enable different
functionality.
If data is meant to be fully functional on another computer system, using the data will be greatly facilitated if
the data is annotated. That can be done by sending data in a standard XML format. Hence, the sender can
try to standardize and enhance the data structure, thus transforming unstructured data into semi-structured
data.
Senders may not like the recipient to use information for unknown or undesired purposes. Hence, they may
transmit data that is not fully functional. The receiver can be hampered because the data is not annotated
and highly unstructured.
Data that is highly structured at the senders site, such as prices of products in a relational database, can be
transmitted in unstructured documents such as highly graphical brochures. An information owner who wants
to prevent poaching and the like (see Section 2.6) may adopt a format that reduces functionality, thus limiting
the value of the transmission for the information receiver (Clemons & Hitt 2004).

The Tradcom case indicates that the data format that companies internally use often matters. This is
especially relevant for companies that limit relationships to short term contracts. Tradcom has a short term
relationship with the customers. Therefore, it allows the customers to send orders in the format they use
internally, such as SAP (Goethals et al. 2007a). This lowers the burden on the customers to do business with
Tradcom. However, not every internally used format is accepted by Tradcom. The platform understands
messages from counterparts in a number of vendor formats (such as SAP). Customers proprietary system
formatted messages are generally not understood (Goethals et al. 2007a). From the supplier-side, where
there are long term relationships, Tradcom-specific XML protocols have been developed. All suppliers have
www.ejise.com
188
Frank G. Goethals
enabled their systems to speak that Tradcom-specific language. Their internal data storage format thus is of
no direct relevance.
Companies can seek to restrict data functionality. As an example, a supplier may not want potential
purchasers to load the entire product catalogue into their ERP systems. This would make it too easy for them
to compare different suppliersprices. Still, it is valuable for the purchaser that data on products he buys from
the catalogue are made persistent in his system. A solution to this problem was defined by SAP and Ariba
(SAP 2003) and implemented at Tradcom. With the Open Catalog Interface (OCI) or punch-out solution, a
purchaser sees the catalogue on a webpage formatted in HTML. The catalogue does not enter the ERP
system of the purchaser. The purchaser can select the products he wants to order via a web interface and
can have a standardized message sent to his ERP system so that only the data on the desired products is
posted to his ERP system.
Many suppliers do not want purchasers to download the entire catalogue into their system because this
would make it too easy for the customers to benchmark offerings. This is, however, not always the case. If
the competitive strength of the company is just that it has the lowest prices, it will try to make its price data as
accessible as possible. In case the supplier tries to differentiate his products from those of competitors using
other variables than the price, the supplier will try to pull the attention of the customers to the appropriate
variables; for example by presenting a nice picture on the catalogue-webpage. Short term buyer-supplier
relationships tend to work this way. Suppliers in longer term relationships may make prices available in an
accessible format, even if these do not show the best side of the supplier.
When considering PLM, for a designer a bit map picture is not very useful: they require vector formats to be
able to make changes to the products design.
2.4 Different parties have to make investments

Investments are needed to enable information sharing. Partners have to agree who will bear the costs for
installing, maintaining and upgrading systems.

Investments can be made primarily by one company or split among the partners and/or third parties. If one
party changes or withdraws the systems he possesses, other parties may have to make further investments
in order to ensure connectivity. Without their investments, information flows will fail if interfaces for example
have become invalid. Long term relationships partners are better placed to know who currently uses a
modified system and are able to negotiate, plan and control changes.
It can be desirable that new entrants to the network do not need to make significant capital investments. If
big investments are required, this complicates the decision to commit to the network because they then are
locked-in to a relationship that may not end in a partnership. On the other hand, a party that leaves can take
a part of the investments with him so that the rest of the Extended Enterprise suffers more than
proportionally from the departure of that party. That is, they lose more than just the connection with that
party: they also lose connection with other parties.
Han et al. (2004) showed companies tend to be reluctant to share information via a non-neutral
infrastructure. The natural fear is that the information will be used beyond a contractual relationship. This
seems significant in cases of coopetition (i.e., cooperation between competitors).
It is thus not only important to decide who pays for the investment just because it negatively influences who
wants to share data (i.e., as it costs money you do not want to share data), but also because it has a positive
influence (i.e., as you own the storage space yourself you are willing to share data).

The spread of investments by collaborators depends on the chosen information sharing configuration.
Therefore, no general comment on VMI or PLM can be given.
In the Tradcom-case (Muylle & Croon 2003), customers did not have to make big investments: they only had
to enable their systems to send purchase orders in their proprietary format to Tradcom. Their suppliers had
to make bigger investments: they needed to communicate in a Tradcom-specific XML format with Tradcom.
www.ejise.com
189
ISSN 1566-6379
Tradcom took care of translating between the customers formats and Tradcoms standardised format. A
group of suppliers together own Tradcom and they are thus the parties that invest most in the
communication. Customers can easily enter and leave the marketplace whereas suppliers are closely tied to
Tradcom. Tradcom reduces investments for individual suppliers because the lowest cost option is to have
suppliers connect only to Tradcom, and this interfaces with systems of all groups customers. If a new
customer enters the marketplace, the individual suppliers do not have to make new investments. If a new
supplier enters the marketplace, other existing suppliers do not have to make investments. Neither do their
customers.
Should one or several of the Tradcom-owning suppliers leave Tradcom and take the Tradcom platform with
them, the customers and all other suppliers would be disconnected.
2.5 Partners become dependent upon service levels provided by data sharing systems
The value of data sharing systems depends on the service levels that are realized.

A significant service level requirement is availability. Lack of data inhibits the functioning of companies.
Users service level expectations have to be managed so that unexpected temporary events that cause
lower availability do not create negative sentiments (Eliadis 2007).
Other service level issues that require attention are system response times, the variability in response time,
the maximum time span specific data is inaccessible, etcetera (Hartley 2005). Current embodied solutions
to deal with contingencies, such as calling a colleague in the partnering company, can become less evident
once the computer systems are integrated. This is because computer systems integration can weaken the
personal relationships that exist between companies (Levina and Vaast 2006). This increases the
dependence upon sustained peak performance of the computer systems.

In the context of VMI, unavailability of the resellers stock and sales data prevents the seller from accurately
defining the purchase order quantities (i.e., the sales order quantities from his viewpoint). The network may
be able to function without this information, but only for a limited time span.
Similarly, companies sharing product data can usually execute their tasks if they have access to not-sorecent product data. Still, sometimes it is important an update is considered in the execution of a task. The
infrastructure then should allow the timely communication of this update. Product data files can be quite
large. Files of more than 100 MB are common. If such a file is suddenly needed, the requesting system may
have to wait a long time for the transmission to complete, disrupting the systems execution.
2.6 Partners must preserve the value of data sharing activities

The partners should handle data in line with how other partners would like them to handle it so that they can
build trust.

There are two main situations that create trust between companies: the receiver has faith in the quality of the
data and the sender has faith in the receiver preserving the confidentiality of the data. This trust can be
marred by the partner himself, or by a third party. This is shown in Figure 1.
Sender (S)
Receiver (R)
Himself (H)
Quality
Poach
Third party (T)
Corrupt/Steal
Steal
Figure 1: Sender and receiver are both responsible for preserving trust
Receivers have several responsibilities. A party that obtains new data has the power to poach. According to
Clemons and Hitt (2004, p94), poaching comprises three components: (1) the exchange of information
between two parties, as a natural by product of contractual exchanges for other goods or services,
www.ejise.com
190
Frank G. Goethals
necessary for the performance of contractual obligation; (2) the subsequent use of this information by the
receiving party, outside the purposes for which the information was provided, and for its own benefit or
economic gain; and (3) at the expense of, or creating economic damage to, the party that provided the
information. By extension, poaching cannot only concern production data, but also meta-data: partners can
pass on data about what data is being shared.
Going one step further, receiving parties may reject poaching but inadequately secure their systems so that
unauthorized parties can gain access to the data. In essence, a network is only as secure as its weakest
module or connection. Security requires authentication and authorization (Logan 2004). Authorizations
typically are discussed in the context of who is trying to access the data content. Goethals et al. (2006a)
found authorizations tend to depend on what task the data will be used for, the physical location the
message has to be sent to, the medium over which the message is transmitted, the moment the message is
sent, the data content, the message format, whether it concerns a single record or a batch of records,
whether the data is coarse-grained or fine-grained, etcetera. Authorizations are thus a complex matter.
Both the receiving and sending parties systems should be trustworthy. The sending party should make sure
that it provides data of the expected quality. Therefore, this party has to prevent stored data corruption and
ensure accurate data entries. This can be a significant task for operators that do not know or understand the
relevance to users.
A special problem concerning the preservation of the functionality of the data has to do with non-repudiation.
Typically, a party executes tasks in response to received messages. Parties that have taken part in
information transmissions should not deny their participation later. In a B2B context, the concept of nonrepudiation is important. This concept embraces two ideas: the sender cannot deny that he sent the
message and the receiver cannot deny that he received the message (Kremer et al. 2002).
Privacy is an often mentioned topic in the context of partnering companies. Clarke (1999) asserts
information privacy requires that information about individuals should generally not be available to others and
that, where such data is possessed by another party, the individual should be able to control the data and its
use to a considerable extent. This leads to the conclusion that a customer that gives personal information to
a specific company, not automatically agrees this information can be forwarded to partners of this company.
Partners clearly should handle data with care, fully respecting the agreement the originating organization has
with its customer.
As the partner does not have a direct relationship with the customer, chances of poaching are elevated.
Poaching can be limited by using a data format with limited functionality (Clemons & Hitt 2004). Logically,
one would not expect poaching to show up in an Extended Enterprise context as poaching potentially
damages trust. Still, Clemons and Hitt (2004) state actual poaching is difficult to observe, turning it more
plausible. They assert poaching is more likely to turn up if there is weak intellectual property protection (e.g.,
the impossibility to return the information at the end of the contract) and if poaching was not prohibited in a
contract. These are two conditions that often apply in a close partnership.
One could make a similar assumption about the problem of securing the systems. One could assume that
partners ensure the security of their systems because they are dealing with confidential information. In one
study, Dynes et al. (2005) indeed found firms do not formulate big security requirements for their suppliers.
Yet, in an earlier study Dynes (2004) found that, in general, companies are auditing the information security
status of potential partners. One difficulty with such assessments is that they are said to slow down
partnering.

Applying Figure 1 to the PLM example suggests the following risks:
(RH) The party can consciously share his partners designs with a competitor of that partner.
(RT)
If a partys systems are not secured well enough, intruders may gain access and steal the product
designs that are owned by this partys partner.
(SH) For one party it can suffice that data are generally precise, while for another party extreme accuracy
may be relevant. Rounding done by the former party (implicitly) results in inaccurate data for the latter party.
(ST)
If an external party can hack the systems and can change measurements, the plans become
worthless. If such problems re-occur this makes it hard for companies to have confidence in the data that is
being shared.
www.ejise.com
191
ISSN 1566-6379
Similar risks exist in the VMI case.
2.7 Data ownership has to be unambiguous

A data owner has to be appointed. A party decides what can or has to happen with the data and what
cannot. Often it is not clear who is the designated data owner.

On the one hand, several parties may want to decide what can or must happen with the data. On the other
hand, no party may be designated as data owner or may feel responsible.
Data ownership is defined in the California State University Data Warehouse Glossary as the responsibility
for determining the required quality of the data, for establishing security and privacy for the data and
determining the availability and performance requirements for the data. Data owners have the authority,
accountability and responsibility to create and enforce organizational rules and policies for business data.
They have the right to determine access, creation, standardization and modification rights (Alstyne et al.
1994). If ownership is not well defined, data that is considered to be valid may in fact not be valid; may fall in
the wrong hands, etcetera.
Levina and Vaast (2006, p28) indicate that, in community-like practices, boundary objects are co-produced
and represent negotiated outcomes of a joint practice. If different partners collaboratively change a view on
an object, it becomes obscure who actually owns that specific view on the object. As a result, the ownership
of intellectual property is one of the most critical legal risks that confront Extended Enterprises (Bader 2008).

If a subpart of a product is created by a partner it is not always clear who is the owner of the subpart-data:
the producer of the final product, or the designer of the subpart. Similarly it may be unclear who the owner of
the stock data is in case of VMI. Unless consignments are used, the stock is the property of the reseller. This
would suggest the reseller is the data owner. The stock is, however, managed by the seller. So, the seller
could be seen as being the data owner. The fact that the reseller and the seller have to agree on a stocking
plan, inventory turns, replenishment rates, delivery frequency and the like shows that both parties have
something to say about the stock. The key issue here is defining responsibility, accountability and authority.
Data ownership problems also show up in cases where there are data that only exist at the level of the entire
Extended Enterprise (i.e., at the level of the collection of the collaborating legal entities). In Tradcoms case,
suppliers sell their products through Tradcom to a multitude of customers. Although individual suppliers only
get orders with respect to their own products, there is information available at Tradcom-level about orders
that contain products from different suppliers and about customers that entered the platform to do business
with one supplier but in time started doing business with other suppliers. The question is who owns this data
and has responsibility for managing its quality. This information can be very valuable yet easily overlooked
because it is by nature not really owned by any of the individual parties.
2.8 The involved parties change over time

All of the problems above have to be dealt with in the frame of changing relationships. Partners can be
added to and can be removed from the network.

The parties that provide data and the parties that use data change over time. Data that is valid changes if
partners change. Also, there are changes in the agreement on which parties can access data: some parties
can probably no longer access the data.
Partners that are dropped from the Extended Enterprise are likely to become competitors of the Extended
Enterprise. Retained access to the Extended Enterprise data may hamper the competitive position of the
Extended Enterprise. However, it can be desirable that they still get access to specific data, for example to
offer after-sales service. Similarly, it is often important for members of the Extended Enterprise to maintain
access to former partners data.
www.ejise.com
192
Frank G. Goethals

When a supplier of a semi-finished product is replaced, the valid product data set typically also changes.
That is, in the design of the assembled product, the design of the semi-finished product has to be replaced
by the design of comparable items made by the new provider, or at least its interface. While the old design
data may not be valid for newly produced products it remains useful to maintain old products and the aftersales service provision. This implies that different versions of the component design have to be managed
and related to specific variants of the produced product.
In general, the former supplier of the component no longer should have access to the other product data as
they could disperse this data across the new network they have entered. Nevertheless, if this supplier still is
to create spare-parts for the product under consideration, they shall need to be given access to specific parts
of the data warehouse.
With VMI the seller truly obtains insight in the resellers business (Monitor Technologies 2005). This can be
illustrated with the fact that if a transmission of sales and stock data fails, the seller, in practice, still has quite
a good idea of the sales and stock levels at the resellers site because he knows his business so well. This
fact is very desirable while both companies do business with each other. However, once their relationship is
suspended, the seller could abuse the knowledge gained from this relationship.
3. Discussion: the problems are important when evaluating data sharing

configurations
The eight problems presented above should be taken into consideration when deciding upon an interorganizational data integration configuration. Many configurations exist that have radically different
properties in terms of how the eight problems show up.
According to Goethals et al. (2007b), one can distinguish between different data integration configurations on
the basis of
1. how decisions with respect to the data sharing are made: decisions with respect to data sharing can be
taken centrally or decentrally. A central consortium such as RosettaNet can define a global data model,
what messages can be exchanged, etcetra. Alternatively, parties can bilaterally agree on the content that
will be shared, on an XML Schema they shall use, etcetera.
2. how data is transmitted: Data can be transmitted directly from data provider to data receiver or data
transmissions can happen via a central system. Such a central system then can take care of format
mappings, guaranteed delivery and other services.
3. how data is stored: The data that is to be shared may be stored in one central repository that is shared
by all partners, or can be stored in different locations of different partners.
A data integration configuration is characterized by the extent to which it is rather centralized or rather
decentralized on these three dimensions. Depending on the degree of (de)centralization, a configuration has
different properties in terms of the problems presented in Section 2. Each configuration has good points and
bad points to note for each problem.
In CIMdata (2005), an entirely centralized data integration configuration is presented where companies
create a single shared data repository that contains all the data the companies want to share. This
configuration can be evaluated in terms of the problems presented above. For example, it was stated that
timing, content and responsibilities for transmissions as well as data access rights have to be determined.
When deciding upon necessary updates of the data in the shared storage space and access rights, a
completely centralized configuration has the disadvantage that an update to the central data is relevant for
one party, but may need to be hidden for another party. Also, for all information requirements that should be
fulfilled there is this third party, the shared storage space, which needs to be involved. This central party can
have established policies that are considered rigid and inappropriate by local offices. This may reduce
motivation to introduce new information flows (Streeter 1973). However, the fact that there is an overall view
available on what information sharing is happening makes it easier to manage the whole system.
As another example, when it comes to the data format, the intermediary can take care of a big number of
tasks, among which the transformation of message formats and the transformation of content. The product
number used in one system, for example, can be reformatted to suit product numbers in another system.
However, adapting the functionality of the data may be a difficult matter in this setting. That is so, not only
www.ejise.com
193
ISSN 1566-6379
because it is harder for a central office to understand local functionality requirements and restrictions, but
also because the data can be available in the commonly owned shared space in a fully functional form.
Furthermore, when considering service levels, there is a single point of failure (Iamnitchi 2000). All parties
use the capacity of that same central point and the capacity that is available for one party thus depends upon
the capacity used by all other parties. Still, as many parties co-invest in central systems, they may
reasonably expect a high-quality shared space, operated by specialists who know how to intervene in case
of problems. As a final example: as there is only one site with data, there is only one site that needs to be
protected from security breaks (Yu 2000). However, if this security system fails a non-trusted party can
immediately gain access to all data.
Centralized configurations can be contrasted with entirely decentralized data integration configurations,
where partners set up direct data exchanges between the system of the partner that wants to use the data
and the system of the other partner where the data is available. In this case, parties autonomously can
decide what data they want to share and are less concerned with high-level restrictions. Mintzberg, (1992)
considered this to be a stimulus for innovation. However, setting up many point-to-point connections
complicates management of the entire integration effort. Also, parties do not share the capacity of a central
system. This lowers the chance that another party is using some capacity one had planned for. This,
however, eliminates the possibility to use capacity that is currently not used by other parties. Capacity can
thus be available in the wrong place (HP 2003). Furthermore, implementing security policies in a highly
heterogeneous environment is harder (Iamnitchi 2000). Several data stores and transmission systems have
to be secured and it is unclear what the security-level is of the Extended Enterprise as a whole and it is hard
to identify whether anyone is poaching. There is no central intermediary that can oversee the unusual
behavior of a party that are breaking trust.
The illustration above highlights that companies are confronted in different ways with the problems from
Section 2 if they choose another data integration configuration. Consequently companies have to find out
how important a specific problem is to them and consider the identified problems when evaluating different
configurations.
Many configurations exist (Goethals et al. 2007b). The goal of this paper is not to give an evaluation of all
integration configurations. This paper discusses problems that repeatedly have shown up when such
configurations were installed and identified that these problems take on different forms when different
configurations are used. Further research is needed to evaluate different configurations with respect to the
problems discussed in Section 2. Management is advised to consider problems in the context of a specific
configuration they have in mind.
4. Conclusions
This paper discusses eight problems that show up when partnering companies decide to share information.
Each problem occurs in different practical situations, as illustrated with examples from the PLM and the VMI
domains and the Tradcom B2B trading platform.
It is important to deal with each of these problems. It was shown that the problems take a different form in
different Business-to-Business data integration configurations. Companies that need to choose an interorganizational data integration configuration should determine appropriate weights for the problems.
One of the problems concerns the distribution of investments and benefits. The amount of counterpartyspecific investments companies are willing to make depends upon the expected duration of the use of the
investment and thus upon the expected duration of the relationship between the companies. Furthermore, a
data-owner is needed at inter-organizational level which determines who can access what data and those
that may not. The value created from using specific information sharing configurations depends on service
levels that are realized and the way partners handle data. Senders are dependent upon their partners to
reject poaching and to secure their systems. Receivers are contingent upon the senders for transmitting the
data in an appropriate quality. Data format problems and obscurity about appropriate data flows further
complicate the issue.
Acknowledgements
I gratefully acknowledge the help from Prof. David Newlands in writing this paper.
www.ejise.com
194
Frank G. Goethals
References
Alstyne, M. Van, Byrnjolfsson, E. and Madnick, S.E. (1994) Why not one big database? Principles for data ownership,
MIT Sloan School Of Management Working Paper Series 3695-94, p 39.
Angulo, A., Nachtmann, H. and Waller, M. (2004) Supply Chain Information Sharing in a Vendor Managed Inventory
Partnership, Journal of Business Logistics, Vol. 25, No. 1, pp101-120.
Bader, M.A. (2008) Managing intellectual property in inter-firm R&D collaborations in knowledge-intensive industries,
International Journal of Technology Management, Vol. 41, No. 3/4, pp311-335.
Carlile, P.R. (2002) A pragmatic view of knowledge and boundaries: boundary objects in New Product Development,
Organization Science, Vol. 13, No. 4, pp442-455.
Chrisman, N. (1999) Trading Zones or Boundary Objects: Understanding Incomplete Translations of Technical
Expertise, Proceedings of the Society for Social Studies of Science Annual Meeeting, San Diego, October 28-31,
p15.
CIMData (2005) Eurosteps Share-A-space Making your intellectual assets the backbone of your information
management strategy. [online], Eurostep, http://www.eurostep.com/files/Share-A-space_CIMdata_2005.pdf.
Clarke, R. (1999) Internet Privacy Concern Confirm the Case for Intervention, Communications of the ACM, 42, 2,
(February 1999) pp60-67.
Clemons, E.K. and Hitt, L.M., (2004) Poaching and the misappropriation of information: transaction risks of information
exchange, Journal of Management Information Systems, Vol. 21, No. 2, pp87-107.
Danese, P. (2004) Beyond Vendor Managed Inventory: the Glaxosmithkline Case, Supply Chain Forum: An
International Journal, Vol. 5, No. 2, pp32-40.
Dynes, S. (2004) Security and Privacy: at odds with speed and collaboration?, Executive Round Table Series, [online],
Tuck School of Business,
http://mba.tuck.dartmouth.edu/digital/Programs/CorporateEvents/SecurityAndPrivacy/Overview2.pdf p. 13.
Dynes, S., Brechbhl, H. and Johnson, M.E. (2005) Information security in the extended enterprise, Proceedings of the
4th Workshop on the Economics of Information Security, Cambridge, MA.
Eliadis, H. and Rand, A. (2007) Setting expectations in SaaS: The importance of the service level agreement to SaaS
providers and consumers, [online], Software & Information Industry Association,
http://www.siia.net/Software/pubs/saas_0207.pdf, p. 14.
Goethals, F. G., Vandenbulcke, J., Lemahieu, W., Snoeck, M. and Cumps B. (2005) Two Basic Types of Business-toBusiness integration, International Journal of E-Business Research, Vol.1, No. 1, pp1-15.
Goethals F. G., Lemahieu W., Snoeck M. and Vandenbulcke J. (2006a) Dependencies between Data Decisions in the
Software Process, Proceedings of the SPW/ProSim Workshop, Shanghai, May 20-21, 2006, LNCS Volume 3966,
pp48-55.
Goethals, F., Lemahieu, W., Snoeck, M. and Vandenbulcke, J. (2006b) Management and Enterprise Architecture Click,
Information Systems Frontiers, Vol. 8, pp67-79.
Goethals, F. G., Lemahieu, W., Snoeck, M. and Vandenbulcke, J. A. (2007a) The data building blocks of the enterprise
architect, Future Generation Computer Systems. Vol. 23, No. 2, pp269-274.
Goethals, F. G., Snoeck, M. and Lemahieu, W. (2007b) The Business-to-Business Integration solution space, Pre-ICIS
Workshop on E-Business, Montreal-Canada, December 9, 2007.
Han, K., Kauffman, R.J. and Nault, B.R. (2004) Information exploitation and interorganizational system ownership,
Journal of Management Information Systems, fall 2004, Vol. 21. No. 2, pp109-135.
Hartley, K.L. (2005) Defining effective service level agreements for network operation and maintenance, Bell labs
technical journal, Vol. 9, No. 4, pp139-143.
HP Invent (2003) Architectures for storage consolidation, [online], HP,
http://h30046.www3.hp.com/uploads/whitepapers/Architectures_for_Storage_Consolidation_whitepaperF.pdf.
Iamnitchi, A. (2000) Centralized vs. Decentralized design in Internet-Scale Applications, Proceedings of the
International Conference on Parallel Processing.
Kremer, S., Markowitch, O. and Zhou, J. (2002) An intensive survey of fair non-repudiation protocols, Computer
Communications, Vol. 25, No. 17, pp1606-1621.
Levina, N. and Vaast, E. (2006) Turning a community into a market: a practice perspective on information technology
use in boundary spanning, Journal of Management Information Systems. Vol. 22, No. 4, pp13-37.
Ljung, R., Johansson, M. and Rosn, J. (2004) Sharing product data through life in the Extended Enterprise, Product
Data Journal, Vol. 11, No. 2, pp40-42.
Logan, C. (2004) Security principles: identity, authentication, authorization, Current IS&T Issue, Vol. 20, No. 2. pp 5.
Mintzberg, H. (1992) Structure in fives, Prentice Hall, p 312.
Monitor Technologies (2005) Remote and Vendor Managed Inventory Management Systems for Bulk Solids, [online],
Monitor Technologies, www.monitortech.com/webseminar/RMI_V2doc.pdf
Muylle, S. and Croon, E. (2003) Venturing into e-commerce in the MRO market, ECCH, p 19.
Nonaka, I. (1994) A dynamic theory of organizational knowledge creation, Organization Science, Vol. 5, No. 1, pp14-37.
Neef, D. (2001) E-procurement, Prentice Hall Upper Saddle River, p 1996.
SAP (2003) Open Catalog Interface (OCI) Release 4.0, [online], SAP,
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/com.sap.km.cm.docs/library/icc/Open%20Catalog%20Interface.p
df.
Sheu, R.K., Chang, Y.S. and Yuan, S.M. (2001) Managing and sharing collaborative files through WWW, Future
Generation Computer Systems, Vol.17, No.8, pp1039-1049.
www.ejise.com
195
ISSN 1566-6379
Smaros, J. Lehtonen, J.M., Appelqvist, P. and Holmstrm, J. (2003) The impact of increasing demand visibility on
production and inventory control efficiency, International journal of physical distribution & logistics management. Vol.
33, No. 4, pp336-354.
Snoeck, M., Lemahieu, W., Goethals, F., Dedene, G. and Vandenbulcke, J. (2004) Events as atomic contracts for
application integration, Data and Knowledge Engineering 2004 Oct; Vol. 51, No. 1, pp81-107.
Star, S.L. and Griesemer, J.R. (1989) Institutional Ecology, Translations and Boundary objects: amateurs and
professionals in Berkeleys museum of vertebrate zoology, 1907-39, Social Studies of Science, Vol. 19, pp387-420.
Streeter, D.N. (1973) Centralization or dispersion of computing facilities, IBM Systems Journal, No. 3, pp283-301.
Takalikar, S. (2004) Leveraging PLM for manufacturing excellence, Express computer, April 2004, [online],
expresscomputeronline http://www.expresscomputeronline.com.
Tang, M., Lee, B.S., Tang, X and Yeo, C.K (2006) The impact of data replication on job scheduling performance in the
Data Grid, Future Generation Computer Systems, Vol. 22, No. 3, pp254-268.
Yu, E. (2000) Centralize or decentralize? A requirements engineering perspective on Internet-scale architectures,
Proceedings of the International Conference on Parallel Processing.
www.ejise.com
196
Theory, Method and Tools for Evaluation Using a Systemsbased Approach

Pivi Jokela1, Peter Karlsudd2 and Martin stlund1
1
School of Communication and Design, University of Kalmar, Sweden
2
School of Human Sciences, University of Kalmar, Sweden
paivi.jokela@hik.se
peter.karlsudd@hik.se
martin.ostlund@hik.se
Abstract: This work introduces an evaluation model for examining organisations that in one form or another depend on
modern information technology for their core activities. The evaluation model, named SUV, is based on a systems
science approach and has been developed at the eHealth Institute (eHlsoinstitutet) at the University of Kalmar.
The central mechanism of the model, the SUV diagram, partitions the scope of evaluation into seven categories and
three levels. The seven categories have been selected on a systems science basis with inspiration from systems
thinking. The rationale for making this the starting point for the evaluation model is that it may be justifiably stated that the
framework provided by systems thinking truly encompasses the breadth of human activity. Making use of this general
framework, the SUV diagram provides a general roadmap to guide the evaluation effort. The three levels (organisation,
technological and individual) were selected to add detail to the analysis complement the categories and enrich the
analysis of each category as well as to the dynamic interplay among them.
The SUV methodology is based on a continuous evaluation process whose driving-force is the wish of the interested
parties to develop their own activities. It is a methodology within whose framework any and all methods for data
gathering deemed appropriate for the evaluation at hand can be used. The use of multiple methods is explicitly
encouraged for the purpose of gaining a multi-perspective view of the organisation/activity under scrutiny.
Based on the accumulated findings from pilot studies, the model was operationalised in the form of an IT application for
electronic surveys. The application is expected to contribute in making the evaluation process more efficient and add
structure to this process.
Keywords: systems science, systems thinking, evaluation methodology, multi-method, electronic surveys
1. Introduction
Taking appropriate strategic or tactical action based on evaluation results is essential for any organisation to
be able to adapt and survive in a changeable environment. However, the process of evaluation is fraught
with problems and its findings are often regarded as incomplete, erroneous or biased (Barroso et al., 2006,
Hartford et al., 2007). One reason is that systems involving human interaction are inevitably messy and
complex. One way to handle this is to divide the system into smaller parts and examine each one at a time.
This is a common tactic to manage complexity, but the downside is that the relationships between the parts
and the ones between the part and the whole are neglected. Systems science provides a framework within
which complex systems can be understood and it offers structure and method to describe and analyze parts
within their context and systems as wholes (von Bertalanffy, 1968, Boulding, 1985, Checkland, 1981). The
usefulness of the systems science view and the holistic perspective in examining organisations has been
both theoretically justified (von Bertalanffy, 1968, Checkland, 1981, Beer, 1979) and is supported by
empirical findings from diverse fields (Sandelin and Sarvimki, 2000, Marine, 2002, Cohen et al., 2007).
The systems science approach can be used to examine a system at any level. Within the boundaries of a
system there exist an infinite number of subsystems (von Bertalanffy, 1968, Boulding, 1985, Checkland,
1981, Edwards, 2005). The process of deconstruction of the system should be guided by the stated purpose
for the study and be regarded as part of defining its scope. This is similar to what Mulej (2007) lables
requisite holism/realism. The inherent complexity of any human system, with its infinite number of
subsystems and complex contexts, implies that any description or understanding of, a system must be based
on incomplete evidence (Backlund, 2002, Jacucci and Hanseth, 2006, Simon, 1969). Furthermore, human
activity is also fleeting, constantly changing and morphing, both at surface level and at a deeper level (Cohen
et al., 2007). Still, all but the most temporary and volatile systems tend to have some kind of stable core,
which contributes to order in the system, although the relationship between structure and order is not fully
understood (Bohm and Peat, 2000).
ISSN 1566-6379
197
Jokela, P., Karlsudd, P. and stlund, M. Theory, Method and Tools for Evaluation Using a Systems-based Approach.
The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 197 - 212, available online at
www.ejise.com
By letting the purpose guide the process of analysis, a useful, albeit incomplete, representation can be
constructed of the system under scrutiny. In practice, this means that some parts of the system will be
analyzed in greater detail while others will be analyzed in less detail. This kind of goal-directed, focused
analysis is necessary to be able to manage the process and also the product of analysis. At the same time
one must not lose track of the whole picture. The marriage between details and the whole (or the wholes) is
the main challenge to be overcome in order to fulfil the potential of a systems science based approach
(Boulding, 1985, Checkland, 1981, Koestler, 1967, von Bertalanffy, 1968). Systems science aids us in this
process by providing a language to describe the systems and their parts; and to provide structure and
guidance in the traversal of systems levels when shifting between different levels of granularity (Cottam et
al., 2005, Edwards, 2005).
The model described here, systematic evaluation also called the SUV model, takes a systems science
based approach to evaluation and uses systems thinking based methods to perform evaluative work and
analyse findings. SUV can be characterised as theory-based evaluation with a holistic approach. SUVevaluation is mainly formative, in a sense that it is aimed to provide evaluation participants with feedback
and to develop new knowledge, which is needed in order to continuously improve the evaluand and its
performance. Involvement of the participants and interactivity between the evaluator and participants are
essential throughout the evaluation process; from the beginning when the diverse evaluation issues and
goals are established to the end when action must be taken to make changes in the organisation, based on
the evaluation outcomes.
SUV has been developed by the authors while working at the e-Health Institute (eHlsoinstitutet), University
of Kalmar. Previous work on the model is described in (Brandt et al., 2003). The SUV tool is especially
suitable for organisations that use (as opposed to having it as their main business) modern information and
communication technology; for example: e-learning (Harun, 2002) and health services (Zhao et al., 2002).
This, of course, applies to most organisations today (in the western world), and in the near future any
organisation that does not, will have a hard time surviving. In the methodology based on the SUV model, the
use of multiple methods is encouraged. One method that is given special attention here is a web-based
questionnaire tool has been constructed to provide technical support and practical guidance in formulating
evaluation questionnaires.
The basic principles of the SUV model may be summarised as follows:
Analysing holistically by systematically considering the parts and their mutual relations.
Analysing activities starting from three levels: Individual, Technology and Organisation, and seven
categories: Goal Seeking, Hierarchy and Relations, Differentiation and Entropy, Inputs,
Transformation Process, Outputs and Regulation.
Enabling previous evaluations to be analysed by the same principles
Within the holistic framework the use of complimentary methods of data capture is encouraged.
The following section will give a brief orientation into the theory on which the model is based. Then follows a
description of the method and the program supporting the model and creating the conditions for evaluation
construction, implementation, analysis and innovation.
2. Evaluating on a systems-based approach

Systems theory is a theoretical approach for formulating general laws for systems, regardless whether they
are physical, biological, social etc. Within organisation theory systems thinking involves that an organisation,
like an organism, is affected by its environment to which it has to adapt for its survival (Boulding, 1985,
Churchman et al., 1969, Litterer, 1973, von Bertalanffy, 1968).
Systems have been defined in the literature in various ways. A frequently quoted specification by Hall and
Hagen (1969) reads A system is a set of components (objects) containing relations between the
components (objects) and between their properties.
A system to be evaluated can be defined as a whole organisation but also as a department or a specific
computer system. It is important to notice that the definition of system boundaries as well as the division into
separate system levels does not follow any strict rules or principles but is a result of the goals and attitudes
of the evaluator (Katz and Kahn, 1980). In an open system information and material are continuously
interchanged with the environment. This continuous interaction with the environment makes it complicated to
www.ejise.com
198
Pivi Jokela, Peter Karlsudd and Martin stlund
distinguish an open system from its surroundings (Checkland, 1981). The aim of the evaluation model is to
facilitate the choice of limitations by using an area division derived from systems theory.
2.1 Main principles for a system

The properties of a system may be explained according to the following basic principles:
1. Holism: A change in part of a system affects the whole system (Boulding, 1985, Litterer, 1973, von
Bertalanffy, 1968).
2. Non-summativity: The whole is more than the sum of the parts (ibid).
3. Equifinality: All open systems can attain the same state or result via different starting-points and
approaches (ibid).
4. Multifinality: A common starting-point and/or the use of different roads may lead to different results (ibid).
5. Circular and manifold causality, non-linearity. A causal connection is not linear but a more refined
explanation model is needed, in that many parts of the system may be affected on different occasions (de
Shazer, 1994).
2.1.1 Holism
Evaluating an activity on a system-theoretical basis entails seeing the whole by systematically studying the
parts and then feeding these results back into the activity from a holistic perspective. Most systems may be
regarded as hierarchically organised, consisting of a long series of subordinate partial systems which all
interact within the superordinate system (Boulding, 1985, Edwards, 2005, Koestler, 1967, Litterer, 1973, von
Bertalanffy, 1968).
2.1.2 Non-summativity
Every partial system is of importance to the whole and should thus be examined separately but also in
relation to the other parts. A hackneyed but relevant expression is that no system is stronger than its
weakest link. Therefore it is essential that all parts of importance to the activity become visible and are
included in the evaluation while at the same time the holistic perspective must be the guiding principle
(Boulding, 1985, Litterer, 1973).
2.1.3 Equifinality and multifinality

In contrast to closed systems, all open systems lack an unambiguously defined point of equilibrium. Activities
and organisations are never static but change more or less continuously. This is the reason why an
evaluation must be considered from a dynamic perspective and why the result must always be regarded as
more or less preliminary. One important concept in systems theory is equifinality, which means that all open
systems can reach the same condition or result from different starting-points and by different approaches. In
other words, an activity may attain the same goal by different methods. Similarly, the principle of multifinality
may be explained as a common starting-point where similar conditions may lead to different results. Even
though an organisation, for instance, gives the same directions and provides the same conditions for its
departments the result may turn out to be completely different (Boulding, 1985, Litterer, 1973, von
Bertalanffy, 1968).
2.1.4 Causality
A cause-effect relation or a linear causality, as it may also be formulated, is common in Western thinking. An
explanation model like this briefly means that all events derive from one cause. This event may in turn give
rise to other events. Linear causality thus means that one cause leads to one event (Petit and Olsen, 1994).
Linear causality is not suitable for systems theory, as it provides a simplified image of reality. In systems
thinking various causality theories have been brought forward, one of the most frequently mentioned being
circular causality. The gist of it is that an event is considered in relation to other events taking place
simultaneously, interfering with each other and affecting each other reciprocally. An individuals actions are
affected by the environment, while these actions simultaneously affect the environment. (Schjdt and
Egeland, 1994), p. 83). The consequence of this is that all events may be regarded as both cause and effect.
All parts or events are dependent on one another (Petit and Olsen, 1994). Circular causality as an
explanation model is not sufficient, however, for studying more complex systems. A more refined explanation
model is manifold causality, according to which it is meaningless to try to find one cause of an event, but that
there may be many causes behind what has happened. If circular and manifold causality are combined, an
www.ejise.com
199
ISSN 1566-6379
even more thoroughgoing and refined causal explanation is attained, entailing that all parts of a system
affect each other directly and indirectly (ibid.).
2.2 Human activity systems

Organisations that can be the target of a SUV-evaluation are human activity systems, in a sense that they
can be modeled as a range of human activities that are related to each other within an organisational
framework and these activities are supported or mediated by information technology. The perception of such
systems is associated with the individual observers image of the world, or according to Checklands
terminology their Weltanschauung (Checkland, 1981). Although these perceptions may sometimes converge
into an agreement of what constitutes a meaningful system, there is no objective method available to create
a joint description of the system. The special nature of the human activity system hinges on the concept that
the system is aware of itself, in the same way that a person is aware of him- or herself. If a member of the
human activity system was to observe the system he/she is part of, this would alter the system.
Consequently, any member of the human activity system is barred from participating in a group observing
this system with the purpose of building absolute, objective public knowledge about the system. However, a
pre-requisite for building public knowledge is that anyone and everyone can participate in observing
something. The exclusion of the members of the human activity systems from observing group makes it
impossible for knowledge about human activity systems to ever reach the level of objective public
knowledge. This implicates that the knowledge that is needed to evaluate human activity systems must be
gained by using a different methodology.
In this work, we postulate that the process of evaluative inquiry can be organised as a learning system,
where multiple issues originating from different Weltanschauung are identified, presented and debated within
a systems framework. This discourse is the basis of shared knowledge as well as new knowledge created
jointly by the evaluation participants, and the deeper understanding may eventually lead to improvements of
the studied organisation. (Jackson and Kassam, 1998, Checkland, 1981)
2.3 The main principles of the evaluation model

In the SUV evaluation model structures and processes of importance are divided into seven categories and
three levels. The categories may be illustrated as the spokes of a wheel, which give stability and direction to
the activity; the levels show the recurring layered structure of an organisation (Figure 1).
Figure 1: The seven categories and three levels in a SUV evaluation.

The three levels of the model can be seen as a nested holarchy, a concept that was introduced by Koestler
(1967) and further developed by Edwards (2005). Each level not only includes but also transcends the
previous levels. The complexity increases successively when one moves towards higher levels (Edwards,
2005). In the SUV model, the fundamental level comprises individuals and interactions between them. At the
next level, information technology is added to the system so that the technology level includes individuals,
www.ejise.com
200
computer systems and networks as well as the human-computer interactions. The third level emphasises
how the individuals and IT-systems are embedded in the organisational structure.
The seven categories are gleaned from work in the field of systems science on the epistemological view of
the world as a system. The categories can, with due justification, be said to encompass the breadth of
human and natural activity, albeit at a high abstraction level (Beer, 1979, Checkland, 1981, Skyttner, 2001,
von Bertalanffy, 1968). For each category an example is given to illustrate the type of issues that are
covered by the category and how they can be operationalised in the form of questions on the personal,
technical and organisational levels, respectively. The examples are from projects at the eHealth Institute.
2.3.1 Goal-seeking
As organisations consist of a number of different parts and hierarchical levels there is often an amalgamation
of goals that have to be defined. The structure of some goals are more overarching and static and will be
relevant to the activity for a long time, whereas others are more mobile and changeable. It is important to
consider organisational goals from different hierarchical levels, as well as making both the formal and
informal organisation goals visible. (Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968).
Project: Evaluation of a web-based customer service Status: Planned
application for 3D-mediated advice-giving on
pharmaceutical products
Personal
What are the personal goals of the pharmaceutical counsellors when using the
service?
Technical
What technical equipment and software will be required for users of the service?
Organisational Can the introduction of this service broaden the range of questions that can be
handled over distance?
2.3.2 Hierarchy and relations

In most organisations it is possible to identify different hierarchical levels and specialised functions.
Elucidating these structures and the distribution of responsibility among them creates important conditions
for the evaluation. Since the holistic perspective is a matter of cooperation and mutual relations between the
parts, the whole cannot build on a collection of autonomous elements. In this part of the evaluation the
external and internal boundaries of the organisation are made clear. (ibid)
Project: Evaluation of a distance learning course for
medical professionals
Personal
Status: Completed
What is the work climate like in the student groups?
Technical
What is the structural relationship between the different technical solutions that are
used in examination with regard to authentication and security issues?
Organisational How are responsibilities and task distributed among the persons involved in running
and administering the course?
2.3.3 Differentiation and entropy

When the parts of an organisation begin to specialise to be able to perform certain tasks, the redundancy
(surplus) as well as the entropy of the system decreases. In the long run the result of this differentiation is
that the various parts become more dependent on each other while simultaneously, to some extent, the total
efficiency of the system increases. A balanced degree of work distribution creates stability and safety but too
much differentiation reduces the flexibility of the organisation and may make reality vulnerable. (Boulding,
1985, Litterer, 1973, Simon, 1969, von Bertalanffy, 1968).
www.ejise.com
201
ISSN 1566-6379
Project: Evaluation of a web-based monitoring system Status: Underway

for home care patients
Personal
What skills do the various professionals who are in contact with the patients have?
Technical
What alternative technical systems are available should the primary system fail?
Organisational What routines are in place for the training of the different categories of
professionals involved in running the system?
2.3.4 Inputs
The input category makes visible the resources and conditions available to implement a certain process
(Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968). This comprises human resources like the knowledge,
competencies and experiences of the individuals involved in the process. It is also important to study the
attitudes and motivations of the people involved, as well as their demands, expectations and fears with
regard to the process. The physical circumstances include the ergonomic design of the work place and the
access to work tools. The IT environment comprises hardware, software and network resources as well as IT
security.
Project: Evaluation of a web-based customer service Status: Planned
application for 3D-mediated advice-giving on
pharmaceutical products
Personal
How many real life birthing sessions has the user participated in?
Technical
What technical capabilities are there in the hardware to sense and react to the
physical actions performed by the midwifery student?
Organisational Is the simulator training co-ordinated with the clinical training?
2.3.5 Transformation process

The transformation process is the actual interaction where the resource inputs are transformed into products
and services (ibid). This kind of interaction may take place between, for example, management employee,
employee customer or human computer.
Project: Evaluation of a health information web site
Status: Completed
Personal
How does the service support the users knowledge learning process?
Technical
In what way do the technical features of the service support the users learning?
Organisational Is the organisation well-adapted to support the purpose of increasing its users
knowledge on health issues?
2.3.6 Outputs
The outputs are a result of the processes taking place in the organisation and are usually synonymous with
the services or products it delivers (ibid). They thus occupy a central position in following-up systems. In an
evaluation it is important to remember that there may be several different goals for what may be considered
a good result. One important question that has to be asked is whether the results created in the evaluation of
the outputs will be effectively fed back into the activity.
Project: Evaluation of a pharmaceutical telephonebased call centre service
Status: Completed
Personal
How satisfied are the clients with the information they have received?
Technical
What level of audio quality is delivered?
Organisational How can the average call time be optimised?
www.ejise.com
202
2.3.7 Regulation
Regulating the parts and process included is necessary for directing the transformation process towards the
goals formulated. Regulation takes place synchronously and continuously on three levels. The first primary
regulation is chiefly based on policy, norms and fixed rules in the organisation and its activities. The second
or secondary regulation may use parts of the feedback occurring in a continuous evaluation in order to
better adapt the contents to external and internal demands. The third adaptation level is attained when the
organisation has been made part of the environment by opening up new paths and goals for its activities. All
the three levels are essential to an open system, because static goals would mean that the system dies of
itself. (Boulding, 1985, Churchman et al., 1969, Litterer, 1973, von Bertalanffy, 1968). The result and the
efforts made around the evaluation may be regarded as the regulation of systems and partial systems,
especially on the second and third levels, which makes the evaluation very important for maintaining the
vitality of the system.
Project: Evaluation of an information exchange
Status: Planned
system for the co-ordination of activities for special
needs kids
Personal
In what way can the system users contribute to the way the service is developed?
Technical
In what way can the features of the system be adapted to suit the individual
preferences of users?
Organisational How are proposals for improvements and changes managed?
2.4 Evaluation research

Evaluation is an important control instrument in all activities, especially in goal-oriented and decentralised
ones, where a clear feedback both to decision-makers and to individuals in the operative activity is required.
There are a number of different models and strategies in the evaluation field, some of which models evaluate
products and others processes (Langerth Zetterman and Lindblad, 2001). Depending on which type of
evaluation model is used, the possibilities of different interested parties to have their needs and interests
elucidated and noticed are affected (Guba and Lincoln, 1989, House, 1980, Madaus and Kellaghan, 2000,
Patton, 1986). It is not uncommon that evaluations focus more on models and methods than on a
theoretically founded research methodology, and frequently criteria set up in advance are applied. In a
comparison with what has been mentioned above the SUV model may be positioned as a theoretically
founded model, which gives the evaluator the freedom to choose and create new criteria in both productand a process-oriented evaluations.
There are many points of contact between evaluation and research (Karlsson, 1999). An essential
overarching similarity is that researchers and evaluators use the same methods for data gathering and
analysis. The term evaluation research is relatively frequently used in describing the application of scientific
research methods in evaluation contexts. However, there are differences between evaluation and research.
One such difference is that evaluation often takes place in a task framework that is narrower than the theory
frame used by researchers. This means that the evaluator is under stricter control than the researcher. A
further difference is that the evaluation task often involves assessment, which is not expected of research.
SUV enables both of these directions depending on which approach is employed.
An evaluation effort based on systems science theory broadens the issue and opens up to a more analytical,
critical and reflective approach. Theory-based evaluation has been recommended and developed by Chen
and his colleagues in reaction to the more product-oriented evaluation tradition. If the theoretical foundation
is too narrow, there is a risk that the approach of the evaluation is confirmatory rather than descriptive and
explanatory (Smith, 1994). Closely related criticism has been vented by Patton (1989), who argues that
theory-based evaluations have gone too far in reducing the focus on and generalisation of causal
connections, criticism that he somewhat tones down in a later work (Patton, 1996). It is important to
emphasise that evaluation according to the SUV model does not focus on causal connections but rather on
pointing to a number of possible influential factors.
The evaluation field contains basically three directions. The first is goal-result evaluation. This means
evaluating towards goals in terms of measurable results. The process is viewed as an implementation of
decisions which need not be motivated. The results are often presented as quantitatively measurable results.
The second direction is process evaluation, which is conducted against criteria for what characterises a
good process. The interest is geared towards the implementation, for instance in order to assess work
www.ejise.com
203
ISSN 1566-6379
methods and contents. The result is assumed to be positive if it can be established that the implementation
corresponds to the requirements agreed on. The third direction is interactive evaluation, which is
characterised by the participation of the interested parties. Questions on the implementation are formulated
on the basis of what different interested parties want to know. The interest focuses on what different
interested groups consider important to evaluate from their various perspectives. Interactive evaluation forms
a pattern developed in the 1980s and 1990s, by which cooperation between different interested parties
increases the relevance of the evaluation questions and results and strengthens their influence (Cronbach et
al., 1981). The goal of the SUV model is to thoroughly involve all the interested parties experiences and
ideas.
Earlier on, the idea prevailed that evaluation meant being able to stay neutral vis--vis the various interests
by making objective measurements of goals which everyone was assumed to share. Today most evaluation
researchers agree that this is not feasible possibly except for narrowly delimited situations or relatively
uncontroversial issues. It is probably more usual to be faced with different views on what should be
examined and against which criteria this should be done. In a situation like this it is important to find forms
which may increase the chances of various groups to make themselves heard. In the SUV model goalseeking is done at an early stage and continues throughout the evaluation process, one important principle
being to view the goals from different system levels. All the individuals involved should be offered the
opportunity to participate in goal-seeking. The object is to be able to make formal and informal as well as
short- and long-term goals visible.
According to Franke-Wikberg (1992), the evaluation aim is formulated primarily from one vertical and one
horizontal perspective. The first vertical evaluation, Type 1, is top-down, while vertical evaluation, Type
2, is bottom-up. Horizontal evaluation takes place on the same organisation level. An example of this is a
personnel groups internal evaluation, where the personnel own the results, deciding among themselves
what they want to convey to other levels within the organisation. To these three levels may also be added a
level outside the organisation consisting of various external interested parties. There is reason to point out
that evaluation according to these different descriptions can function in parallel. However, confusion often
arises or attempts to hold up one type as the only right one (Karlsson, 1999). The SUV model enables
formulating the aim from all of these starting-points, which is often an advantage, since it sheds light on the
issue to be evaluated from different perspectives.
3. Methodology
The SUV methodology is based on a continuous evaluation process whose driving-force is the wish of the
interested parties to develop their own activities. What are in several evaluation methods termed process
and product are studied from several points of view on the basis of a holistic approach. By continuously
viewing the strengths, deficiencies and connections of the evaluated systems major and minor changes may
be carried out continuously.
The five basic principles of the SUV methodology may be summarised as follows:
A holistic approach
All interested parties take an active part in the evaluation
A flexible model enables the use of several different theories and methods
The result of the evaluation is used for improving and developing activities
A continuous evaluation process
The evaluation model described here is valid for use in both quantitative and qualitative approaches and can
be incorporated in inductive and deductive studies. An example of deductive use is to construct a
questionnaire based on the SUV categories and levels. SUV also encourages to building up explanations via
an inductive approach in a continuously explorative process, which to some extent reminds of the theorygenerated approach called Grounded Theory as formulated by Glaser & Strauss (1967). An example of
inductive use is the post factum categorisations of previously performed evaluations not using the model,
see 3.2.
The SUV model is not restricted to any one method for gathering data. It is a methodology within whose
framework any and all methods deemed appropriate for the evaluation at hand can be used. Moreover, the
use of multiple methods, used simultaneously, in parallel or sequentially; is explicitly encouraged for the
purpose of gaining a multi-perspective view of the organisation/activity under scrutiny. A survey tool has
been developed for questionnaire construction, web based delivery of surveys and compilation and analysis
www.ejise.com
204
Methods
Compilation
of results. The survey tool is used exclusively for quantitative studies and while it serves as a good example
of how the SUV principles can be put to practical use, it represents only one out of many methods that can
be used with SUV. The choice of methods depends on the purpose of the evaluation and should be given
careful consideration at the outset of the evaluation project. Figure 2 presents an overview of the conceptual
foundations for the SUV methodology and its inclusive meta-structure within which any suitable method of
data inquiry and any suitable approach for data analysis can be used.
Compilation of results
Observation
Survey
tools
Participatio
n
Interview
...
Complementary
approaches
Metho
dn
SUV
Approach 1
SUV-O
Operationalization
SUV-A
Analysis
Operationalization of a
systems science based
approach
Analysis of results
Approach 2
Systems Science
Approach n
Evaluation
Systems science practice
Systems science theory
Evaluation practice
Evaluation theory
Figure 2: SUV is based on Systems Science Theory and Practice. The figure also illustrates the
independence of method.
Figure 3 describes the process of an evaluation study according to the SUV methodology. The iterative
quality in the process of evaluation is emphasized with the resulting measures and innovations serving as
input in the next cycle of evaluation. In a large scale evaluation project, several parallel processes of the kind
illustrated in the figure can be active at the same time starting and ending at different points in time. Going
backwards and forwards between sub-steps in an iterative fashion will (and should) also occur. All the
processes of evaluation should, however, follow the same general progression starting with introductory
delimitation and problematisation and ending in proposed measures and innovations.
www.ejise.com
205
ISSN 1566-6379
Figure 3: SUV, a continuous evaluation process
3.1 Step 1 Introductory delimitation and problematisation

This step includes an introductory discussion about what is to be evaluated, what is the background of the
task and for what context is the effort planned. Preliminary system delimitation is made in the form of a
sketch of what and who should be included in the evaluation. The evaluator should also formulate the
overarching goal as clearly as possible and make a list of what is considered to be most relevant to the
evaluation. It may also be useful to reflect on evaluators pre-understanding and how it may affect the
planning of the study.
The following questions may serve as guidelines during the introductory phase:
Why should there be an evaluation?
What knowledge is desired?
Which interested parties are involved in the evaluation?
What should to be the role of the evaluator?
How should the evaluation be conducted?
How should the evaluation be utilised?
3.2 Step 2 Deconstruction of previous evaluations

It is possible, and often also beneficial, to feed back previously conducted evaluations into the SUV model,
and these results can then be sorted into and interpreted within the categories the model includes. Even if
this kind of revised categorisation cannot be absolute, it may give indication on how accurately the previous
evaluations have covered the entire problem area, and also on which previous questions may be important
to follow up.
3.3 Step 3 Problematisation and delimitation

The third step in the model is to make a more exact formulation of problems. By determining scope and level
the system will be delimited, the choice of what to evaluate depends on the goal of the task. Anything outside
the system boundaries is defined as the environment and will not be included in the evaluation. However, it
is important to keep in mind that there is a continuous interaction between the system and its environment.
The evaluators list of what is considered to be most relevant to the evaluation (see step 1) can be refined
and completed. Following steps can be used as guidelines in this phase:
www.ejise.com
206
3.3.1 The evaluation question

Formulate the evaluation questions as clearly as possible; both the overarching problematisation and more
specific questions.
3.3.2 Delimitation of the system

Set clear boundaries for the system and specify what is to be included in the evaluation within these
boundaries. It is also important to clarify which individuals are to be involved in the evaluation.
3.3.3 Safeguard the three levels

Make sure that the three levels, individual, technology and organisation, are represented in the evaluation.
3.3.4 Scope and timeframe

Clarify what resources are available in the form of individual effort, time, and access to respondents. Based
on the evaluations scope and available resources, make a time schedule for the evaluation process.
3.4 Step 4 Method choice/Question construction

The SUV method allows for a range of different forms of data gathering, such as observation, focus group,
interview, questionnaire and document analysis. In the same way as the system delimitation, the choice of
appropriate methods depends on the evaluation goal and also on questions that are defined in step 3. Using
various methods in the same evaluation usually makes the study more refined and valid.
The seven categories and three levels in the SUV circle are used as a framework for methodological design
and also for question construction. Preferably, questions should be constructed for every single one of the
fields in the SUV circle. Such full coverage will indicate that the evaluation takes into account all the
important properties that characterize a complex system.
3.5 Step 5 - Dialogue, communicating questions

In SUV model, it is essential that the evaluation questions as well as obtained results are directly
communicated to the participants. To enable a more formative and communicatively oriented evaluation,
asynchronous and synchronous discussion conferences can be used to continually interpret and discuss
findings; this approach can also contribute to the evolving evaluation method.
The web-based questionnaire tool contains several functions to convey questionnaires to respondents.
Among the functions may be mentioned advance notice, automatic reminder and a report function for
processing the gathered data.
3.6 Step 6 Visualisation of data

In this step, the gathered empirical data is compiled and presented in suitable ways, both quantitatively and
qualitatively. The previous categorization of questions into SUV categories and levels, can be used to
structure the results in preparation for their subsequent analysis. One example of such structuring can be
seen in figure 4, where the results from multiple choice questions are presented for each of the fields in the
SUV circle (this functionality is included in the survey tool previously presented).
www.ejise.com
207
ISSN 1566-6379
2.7
3.4
3.2
4.1
3.5
4.2 3.7
3.2
2.8
3.
3.
4.4 3.1
Figure 4: Visualisation of the mean values from multiple-choice questions.
3.7 Step 7 Analysing the result

Assessment or analysis is the central task of evaluation (Karlsson, 1999). The analysis is often the least
common denominator in different definitions of evaluation where the evaluation method constitutes the
framework of the analysis. At the analysis/assessment of the result the evaluator may adopt a more or less
objective or subjective attitude. The objective attitude includes general evaluation principles setting up
norms, while the result in the subjective attitude is more dependent on the interpreter. It is also possible to
choose the dialogically inter-subjective standpoint without getting stuck in objective or subjective reasoning
(ibid.). The analysis may instead be shaped after a discussion between evaluators who try to get an insight
into and understand each others perspective. Consequently, this qualitative analysis takes place in a
dialogue with different interested parties.
Comparison is one normative analysis approach used in evaluation contexts. There are basically three ways
of viewing results. The first is to make a comparison with an explicitly defined reference point of some kind,
such as a goal, idea or theoretical criterion. The second is comparing evaluation objects of the same kind,
usually by ranking. The third is to compare transformation over time, in other words the development of the
evaluation object. In SUV the starting-point of comparisons consists of the categories and levels stipulated in
the model, which enables the use of all three approaches. A comparison emanating from its theoretically
founded variables may be discussed using the following questions as starting-points:
What qualitative differences exist within the various categories and levels?
Which categories/levels received low and high values, respectively?
Between which categories/levels are the differences the greatest or the smallest?
How does the result change over time?
On the basis of this orientation any reasons behind the differences displayed are analysed. It is also possible
to further refine the categories which have indicated deficiencies or those categories which may serve as
positive examples.
3.8 Step 8 Taking action

At the very planning of an evaluation the issue of how to utilise the obtained results should be discussed. On
the basis of the analysis in step 7 an assessment can be made of which categories require further scrutiny or
immediate action to be taken. The SUV model makes it possible to start this process at an early stage and to
continuously adjust and attend to various phenomena and conditions, even during the evaluation process.
The fact that evaluation results are not always utilised for further development of the organisation and its
activities may be due to various reasons. There may be external environment or background circumstances
which affect the structure and implementation of the evaluation. It may also be due to how the evaluation is
conducted and whose questions have been actualised. Further reasons may be that the results of the
www.ejise.com
208
evaluation are not considered relevant or that they are presented in a deficient way or that the intended
receivers lack the knowledge, motivation and resources to operationalize the evaluation results (Karlsson,
1999).
There are various measures that can be taken in order to support the utilisation of the evaluation results. To
start with, it is essential to create good representativeness and maintain a active dialogue with those who are
in various ways involved in the evaluation process. Another important contribution to enhance the positive
attitudes is to emphasise the importance of communication and interaction among the participants, The
results should be reported in a way that makes them available and interesting. In some cases, it may be
important to raise the general competence level in the organisation, for instance by educating personnel and
by encouraging discussion and critical reflexion in practice. An important measure for enhancing positive
attitudes is to set aside resources which enable the application of those evaluation results which are
considered important (Karlsson, 1999).
4. Web-based questionnaire tool

The questionnaire application was constructed by the authors to create a tool for question generation, data
gathering and analysis that would make the SUV method accessible and usable also for the non-systems
theory savvy user. The questionnaire instrument includes the possibility of choosing between different
question types and between different formats, as shown in Figure 5. The questions may be divided into
background questions, SUV questions and open questions.
Figure 5: Different questions types and formats in the web-based questionnaire tool.
www.ejise.com
209
ISSN 1566-6379
Background questions are used in the statistical analysis to see the differences between one group and
another. When searching for possible explanations it is important to consider which variables are relevant to
the current evaluation. It is common to ask questions on sex, age, education, computer experience etc., but
every organisation has its own special conditions which may be of importance to use as background
variables.
The multiple questions provided with a six-grade answering scale are termed SUV questions and are
formulated so that each question belongs to a specific field in the SUV graph. The user marks in which field
the question has been constructed and the tool states the frequency of questions in each field, as shown in
Figure 6. This illustrates the distribution of questions among the different fields and also among the different
categories and levels.
Figure 6: Examples of questions and of how the tool visualises their location and distribution within each
question field.
There is also a possibility for the respondent to add further views on the SUV questions. In questionnaires
distributed in paper form such a possibility is often offered at the end of the questionnaire, whereas the
digital SUV tool makes it possible to attach comments immediately after each SUV question. The advantage
is that this enables catching the spontaneous response, which may otherwise be lost if the respondent has to
wait till the end of the questionnaire.
To create a clear structure for result presentation and analysis according to the model there is a specially
adapted result presentation module in the program. The result presentation is based on the SUV graph and
on the collected quantitative data which is used in the statistical calculations. The arithmetic mean as well as
median value of the answers to the SUV questions can be shown in each in each field, category and level,
respectively. The result is also presented in greater detail by including the frequency distribution within the
various answer alternatives in a table and visualising them in a suitable graph. For background variables
either a bar chart or a histogram is used, depending on whether it is a discrete or continuous variable.
If the number of respondents is sufficient the connection between background variables (groups) and SUV
2
questions may be studied by means of a chi-two test ( ) (Curwin and Slater, 2004). It is important to
emphasise that the statistical analysis is mainly used to looking for possible significant connections in the
gathered material, the primary aim thus not being to draw general conclusions that are supposed to be
relevant to all organisations.
The questionnaire tool also comprises a question archive where one may simply include and edit preconstructed and pre-categorised questions. Usually, questions suitable for elucidating the various categories
can be reused after a slight modification
5. Experiences and further elaboration

Constructing questions with the support of the SUV model does not automatically result in a holistic
evaluation. It is certainly quite possible to use the principles of question construction to a smaller extent, but
for SUV to achieve reliable reality the guideline is that all categories at all levels are well covered, as our pilot
www.ejise.com
210
evaluations demonstrate. It is also a clear advantage if the evaluation is conducted on a number of

occasions and preferably complemented by several different data gathering methods. This complementation
might suitably be performed after the first more explorative study. To continuously measure the status of the
activity according to the same evaluation model enables a longitudinal comparison on the basis of a common
theoretical reference frame.
Formulating questions that cover all categories and levels in the model may sometimes be difficult and
tedious, and it certainly requires more effort of the evaluator. On the other hand, the proposed structure also
provides that all parts of importance to the activity are included in the evaluation, and at the same time the
holistic view of the problem area becomes the guiding principle. In our opinion, a specially adapted model
and methodology are required to fully acknowledge the complexity of organisations where the use of
information and communication technology forms an integral part of core activities.
As mentioned above, the SUV methodology can be used with several different methods. The same broad
approach also applies to the analysis. The result may reflect a number of theories depending on the
perspective/s adopted. A theory discussion is a necessary element if the method is to be used in a strictly
scientific study.
Within the next few years the eHealth Institute is going to use SUV as its evaluation model in the projects
conducted by the institute. The method will most likely be developed further in evaluations of e-related
projects.
Acknowledgements
Thanks to Prof. Gran Petersson at eHlsoinstitutet for his continuing support for the SUV evaluation project.
Our gratitude to the reviewers for their valuable comments.
References
Backlund, A. (2002) The Concept of Complexity in Organisations and Information Systems. Kybernetes, 31(1).
Barroso, J., Sandelowski, M. & Voils, C. I. (2006) Research Results have Expiration Dates. Ensuring Timely Systematic
Reviews. Journal of Evaluation in Clinical Practice, 12(4), 454-462.
Beer, S. (1979) The Heart of Enterprise, New York, John Wiley.
Bohm, D. & Peat, D. (2000) Science, Order and Creativity, London, Routledge.
Boulding, K. E. (1985) The World as a Total System, Beverly Hills, Sage Publications.
Brandt, P., Jokela, P., Karlsudd, P., Rvay, P. & Wennberg, L. (2003) Evaluation model for e-phenomena a case study.
Proceedings of the MicroCAD 2003 International Scientific Conference 6-7 mars 2003. Miskolc, Hungary.
Checkland, P. (1981) Systems Thinking, Systems Practice, New York.
Churchman, C. W., Ackoff, R. L. & Arnoff, E. L. (1969) Analysis of the Organization. IN A, L. J. (Ed.) Systems, Control
and Adaptation Volume II (1980). NewYork, John Wiley & Sons.
Cohen, S., Kaimenaki, E. & Zorgios, Y. (2007) Assessing IT as a Key Success Factor for Accrual Accounting
Implementation in Greek Municipalities. Financial Accountability & Management, 23(1), 91-104.
Cottam, R., Ranson, W. & Vounckx, R. (2005) Life and Simple Systems. Systems Research and Behavioral Science, 22,
413-430.
Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, D. C., Hornik, R. C., Philips, D. C., Walker, D. F. & Weiner, S.
S. (1981) Toward Reform of ProgramEvaluation: Aims, Methods, and Institutional Arrangements, San Francisco,
Jossey-Bass Publishers.
Curwin, J. & Slater, R. (2004) Quantitative Methods, London, Thomson Learning. de Shazer, S. (1994) Words Were
Originally Magic, New York, W W Norton & Company.
Edwards, M. G. (2005) The Integral Holon A Holonomic Approach to Organisational Change and Transformation.
Journal of Organizational Change, 18(3).
Franke-Wikberg, S. (1992) Utvrederingens mngfald: ngra ledtrdar fr vilsna utbildare, Stockholm, FoU-enheten,
UH.
Glaser, B. & Strauss, A. L. (1967) The discovery of grounded theory: Strategies for qualitative research, Chicago, Aldine.
Guba, E. G. & Lincoln, Y. S. (1989) The Fourth Generation Evaluation, Newbury Park, Sage Publications.
Hall, A. D. & Hagen, R. E. (1969) Definition of System. IN Litterer, J. A. (Ed.) Systems, Control and Adaptation, Volume II
(1980). NewYork, John Wiley & Sons.
Hartford, K., Carey, R. & Mendonca, J. (2007) Sampling Bias in an International Internet Survey of Diversion Programs in
the Criminal Justice System. Evaluation & the Health Professions.
Harun, M. H. (2002) Integrating e-Learning into the Workplace. The Internet and Higher Education, 4, 301-310.
House, E. R. (1980) Evaluating with quality, London, Sage Publications.
Jackson, E.T & Kassam, Y. (1998) Knowledge Shared: Participatory Evaluation in Development Cooperation, Ottawa,
IDRC/CRDI.
www.ejise.com
211
ISSN 1566-6379
Jacucci, E. & Hanseth, O. (2006) Taking Complexity Seriously in IS Research. Information Technology and People,
19(1), 5-11.
Karlsson, O. (1999) Utvrdering mer n metod. Tankar och synstt i utvrderingsforskning, Stockholm, Norstedts
tryckeri.
Katz, D. & Kahn, R. L. (1980) The Definition and Identification of Organisations. IN A, L. J. (Ed.) Organizations, Structure
and Behavior. NewYork, John Wiley & Sons.
Koestler, A. (1967) The Ghost of the Machine, London, Arkana.
Langerth Zetterman, M. & Lindblad, S. (2001) Learning about e-learning. A strater about Internet discourses and
borderless education. NFPF:s 29. congress. Pedagogikens mngfald. Lrande infr och utanfr institutionerna.
Stockholm, Sweden.
Litterer, J. A. (1973) The Analysis of Organizations, New York, John Wiley & Sons.
Madaus, G. M. & Kellaghan, T. (2000) Models, Metaphors and Definitions. IN Stufflebeam, D. L. (Ed.) Evaluations
Models. Viewpoints on Educational and Human Services Evaluation. Hingham, MA, USA, Kluwer Academic
Publishers.
Marine, R. J. (2002) A systems freamework for evaluation of faculty web-work. New Directions for Institutional Research
year, 63-72.
Mulej, M. (2007) Systems Theory: A Worldview and/or a Methodology Aimed at Requisite Holism/Realism of Humans
Thinking, Decisions and Action. Systems Research and Behavioral Science, 24, 347-357.
Patton, M. Q. (1986) Utilization-Focused Evaluation 2nd edition, London, Sage Publications.
Patton, M. Q. (1989) A context and boundaries for a theory-driven approach to validity. Evaluation and Program
Planning, 12 (4), 375-377.
Patton, M. Q. (1996) Utilization-Focused Evaluation: the new century text, Thousand Oaks, CA, Sage Publications.
Petit, B. & Olsen, H. (1994) Om svar anhlles. En bok om interaktionistiskt frndringsarbete Andra reviderade
upplagan, Stockholm, Mareld.
Sandelin, B., S & Sarvimki, A. (2000) Evaluation of patient-focused health care from a systems perspective. Systems
Research and Behavioral Science, 17:6, 513-525.
Schjdt, B. & Egeland, T. (1994) Frn systemteori till familjeterapi, Lund, Studentlitteratur.
Simon, A. H. (1969) The Architecture of Complexity. IN A, L. J. (Ed.) Systems, Control and Adaptation Volume II (1980).
NewYork, John Wiley & Sons.
Skyttner, L. (2001) General Systems Theory. Ideas & Applications, Singapore, World Scientific.
Smith, N. L. (1994) Clarifying and expanding the application of program theory-driven evaluation. Evaluation Practice,
15(1), 83-87.
von Bertalanffy, L. (1968) General System Theory, New York, George Braziller.
Zhao, Y., Yagi, Y., Nakajima, I. & Jujozi, H. (2002) IP Telephony New Horizon for Telemedicine and e-Health. Journal
of Medical Systems, 4(4), 277-283.
www.ejise.com
212
Evaluating Service Quality Dimensions within e-Commerce

SMEs
Graham D. April and Shaun Pather
e-Innovation Academy, Cape Peninsula, University of Technology, Cape Town, South Africa
gdapril@gmail.com
pathers@cput.ac.za
Abstract: With the continued growing investment in WWW technologies by e-Commerce businesses the measurement
of Information Systems (IS) effectiveness in this business sector has become increasingly important over the last
decade. As business users, especially in the SME sector, have become reliant on outsourced IS service providers for a
wide range of services, the quality of service rendered by the latter is an important issue which impacts on IS
effectiveness. Researchers have since the 1990s recognised the importance of service quality as a measure of IS
performance. The literature suggests that IS service delivery to e-Commerce businesses needs to be evaluated
differently to that of traditional brick-and-mortar businesses. There is however a paucity of research regarding IS
evaluation in e-Commerce environments, including that of the application of service quality principles. It is thus difficult
for managers of IS service providers in this context to develop a complete picture of the effectiveness of the IS they
deliver.
This paper reports on a study which investigated whether IS service quality criteria and dimensions applied in large brickand-mortar organisations, are also applicable to SME e-Commerce businesses in the tourism sector in South Africa. In
pursuit of this objective an IS-adapted SERVQUAL instrument was tested in an e-Commerce SME environment. The
research results indicate that, although SERVQUAL principles are applicable to the e-Commerce SME context, the
service quality dimensionality is different. The research derived four new dimensions for service quality expectations of eCommerce SMEs viz., Credibility, Expertise, Availability and Supportiveness. A fifth dimension is the Tangibles
dimension, which is retained from SERVQUAL. Furthermore the results indicate that the Credibility dimension was the
most important dimension in this research context, while the Tangibles dimension was the least important.
Keywords: information systems, evaluation, e-commerce, WWW, service-quality, SME, SERVQUAL, IS outsourcing
1. Introduction
Over the past 25 to 30 years researchers have developed several approaches to evaluating Information
Systems (IS) effectiveness, e.g. IS usage, user information satisfaction (UIS), quality of decision-making,
productivity from cost/benefit analysis, and system quality (Pather et. al. 2004). Since the 1990s IS
researchers introduced a new perspective to IS effectiveness measurement, namely Service Quality. The
latter has subsequently been recognised as an important performance metric in the delivery of IS (Pitt et al.,
1995; DeLone & McLean, 2003; Kettinger & Lee, 2005).
However, IS research in the area of service quality, has focused mainly on traditional brick-and-mortar
organisations in which there is usually an in-house IS department or function. Many authors have empirically
researched IS service quality as a measure of the performance of this function (e.g. Pitt et al., 1995; Van
Dyke et al., 1997; Pitt et al., 1997; Kettinger & Lee, 1997; Watson et al., 1998; Kang & Bradley, 2002; Jiang
et al., 2002; Bharati & Berg, 2003). However, an IS literature search revealed no such studies in an e1
Commerce context . That is, no empirical research could be found which investigates the performance of IS
service providers when servicing e-Commerce businesses. This concern has been substantiated
Studies in the e-Commerce environment have focused mainly on the service link between the e-Commerce
business and its end-customers, i.e. website users. Furthermore, Small and Medium Enterprises (SMEs)
generally outsource their IS functions in order to access the necessary IS expertise (Al-Qirim & Bathula,
2002) which are generally not available within the business. Thus external IS service providers are usually
employed by these SMEs. The high reliance of these SMEs on outsourced IS service providers for the
support of business critical systems implies that their service expectations could be different from those in a
large brick-and-mortar organisation. Furthermore, IS service quality expectations of business units within a
large organisation are influenced by issues such as corporate culture and standardised service levels
agreements. However there are no such influences when an IS provider has a diverse number of SME
clients, each with individualised expectations of service quality. Consequently, IS providers should be
enabled with a firm understanding of service expectations of their clients in the SME e-Commerce
businesses environment.
1
This concern is also substantiated by DeLone & McLean (2004), Pather et. al. (2004) and Hong & Zhu (2006).
ISSN 1566-6379
109
April, G. D. and Pather, S. Evaluating Service Quality Dimensions within e-Commerce SMEs. The Electronic Journal
Information Systems Evaluation Volume 11 Issue 3 2008, pp. 109 - 124, available online at www.ejise.com
In light of the foregoing, the research reported on in this paper examines the IS service quality expectations
of SME e-Commerce businesses i.e. the clients of IS service providers. The main research objective was to
investigate and determine service quality dimensions for this business environment. The principle research
question of the study was: What are the Service Quality dimensions for the evaluation of service
expectations of e-Commerce SMEs?
2. Literature review
2.1 The e-Commerce SME research context
The case for the research question has been made by a combination of issues raised in different studies.
Firstly, Molla and Licker (2001) argue that further research is required to investigate whether IS effectiveness
measurement in the e-Commerce context, should be approached differently to that of traditional IS.
Secondly, the literature also suggests that the application of e-Commerce in small businesses is different
from that of large businesses (Lui & Arnett, 2000; Stansfield & Grant, 2003). Although the principles of eCommerce trading are the same regardless of the size of the business, differences arise especially with
regards to how IS are managed. In-house IT units in large businesses are generally able to support and
manage IS, but this is not the case for many medium, and almost all small and micro enterprises. According
to Rohde (2004), SMEs also generally outsource their IS functions to external IS service providers.
Consequently, the level of service quality delivered by these service providers is considered a critical
success factor for SMEs (Kim et. al., 2003). Together, all these factors suggest that empirical research of IS
service quality in the context of e-Commerce SMEs, would be a useful contribution to the IS effectiveness
research field.
2.2 IS effectiveness research

The measurement of IS effectiveness has been widely discussed in the IS literature, and has been a longstanding concern for both academics and IS practitioners (Grover et al., 1996). IS effectiveness can be
defined as the extent to which a system achieves the goals for which it was designed (Lui & Arnett, 2000).
However, in todays competitive world, IS are also expected to contribute to achieving the organisations
mission, improve productivity and facilitate service delivery (Elpez & Fink, 2006).
Researchers have had difficulty finding appropriate metrics to measure IS effectiveness (Pather et al., 2004),
and many researchers have resorted to surrogate measures (Elpez & Fink, 2006). From the multitude of IS
effectiveness measures, Grover et al. (1996) identified some of the more prominent measures used in the
literature, viz., IS usage, user information satisfaction, quality of decision making, productivity from costbenefit analysis, and system quality.
DeLone and McLean (1992) developed a framework for classifying the multitude of effectiveness measures
into six categories. They called this framework the DeLone and McLean (D&M) IS success model (Figure 1).
DeLone and McLeans research brought about some structure to IS effectiveness research (Seddon & Kiew,
1996).
Figure 1: D&M IS Success Model (Source: DeLone & McLean, 1992)
www.ejise.com
110
Ten years after the publication of their original IS Success Model, DeLone and McLean (2003) reviewed
more than 100 journal articles dealing with IS success measurement. They subsequently revised their
model, producing the Updated DeLone and McLean IS Success Model (Figure 2). Some of the revisions to
the original model were based on suggestions and re-specifications from other researchers in the field.
Figure 2: Updated D&M IS success model (Source: DeLone & McLean, 2003)
Importantly, amongst various amendments to the IS success model in 2003, was the introduction of a
service perspective. IS effectiveness researchers up to the mid 1990s have shown bias towards a product
perspective of evaluating effectiveness, while ignoring the service-based perspective (Whyte & Bytheway,
1996; Lomerson & Tuten, 2005). One exception was the study conducted by Pitt et al. (1995), whose
research was the basis for DeLone and McLean (2003) adding the Service Quality measure to their updated
model (Figure 2). Pitt et al. (1995) believed that the prominence of the service-based dimension had
increased since the advent of the personal computer (PC). They assert that the PC had resulted in more IS
users interacting with the IS department more often. Wilkin et al. (2001) offered two ways of interpreting how
service applies to the IS function. Firstly, an IS can be seen to be more than just a technical product, it can
also generate value from its capacity to serve the needs of its end-users/stakeholders (Wilkin et al.,
2001:113). Thus the whole system provides service to the stakeholders by serving their needs and providing
pertinent information.
The second view of service deals with the service or support delivered by the IS department or external
service providers. Examples of support tasks that IS users expect the IS department to assist them with
included hardware and software selection, installations, problem resolution, connection to LANs, systems
development and software education (Pitt et al.,1995).
2.3 Service quality as an IS performance measure

The foundation of Service quality research in the IS literature is rooted in the work conducted by researchers
in the marketing field. In the marketing literature the authors Parasuraman, Zeithaml and Berry (1985, 1988,
1991, 1993, 1994a, 1994b) have been particularly influential. These authors developed a measuring
instrument called SERVQUAL, to measure service quality from the customers perspective. The SERVQUAL
scale comprises of five service quality dimensions (Table 1) and 22 items.
Table 1: SERVQUAL dimensions (Adapted from Parasuraman et al., 1988)
Dimension
Reliability
Responsiveness
Assurance
Empathy
Tangibles
www.ejise.com
Dimension Meaning and Attributes

Ability to perform the promised service dependably and accurately
Willingness to help customers and provide prompt service.
Knowledge and courtesy of employees and their ability to inspire trust
and confidence.
Caring, individualised attention the firm provides its customers.
Physical facilities, equipment, and appearance of personnel.
111
ISSN 1566-6379

Although SERVQUAL had its detractors (e.g. Cronin & Taylor, 1994) 2 , Parasuraman et al. (1988:31)
believed that the SERVQUAL instrument could be adapted or supplemented to fit the characteristics or
specific research needs of a particular organisation.
Pather et al. (2004), having conducted an extensive literature review in this area, found that Kim (1990) was
one of the first IS researchers to introduce the service quality perspective to IS user satisfaction research.
The appropriateness of the SERVQUAL instrument for the IS domain was subsequently researched by
authors interested in measuring user satisfaction of the IS department. Watson et al. (1993) proposed that
the SERVQUAL scale be adapted to measure IS service quality as a surrogate measure for user
satisfaction. Pitt et al. (1995) subsequently tested the use of the SERVQUAL scale empirically across three
IS organisational types. They concluded that the study provides evidence that practitioners can, with
considerable confidence, use SERVQUAL as a measure of IS success (Pitt et al., 1995:182).
However, Van Dyke et al. (1997) challenged the assessment of Pitt et al. (1995), and questioned the validity
and usefulness of SERVQUAL in the IS domain. In response to this critique, Pitt et al. (1997) criticised Van
Dyke et al. (1997) for using arguments without empirical backing and for not offering viable alternatives.
They re-iterated that the developers of SERVQUAL used rigorous empirical research in their development of
the model
Kettinger and Lee (1997) in extending the debate agreed with the counter-arguments of Pitt et al. (1997).
Kettinger and Lee (1997) adapted the 22-item SERVQUAL instrument for the IS environment, to reflect IS
specific issues such as software, hardware and computer technology. After empirically testing the adapted
scale in the IS domain, the authors used factor analysis techniques which resulted in a condensed 13-item
scale called IS-SERVQUAL which omitted 9 of the original SERVQUAL items and had only four dimensions
(the original Tangibles dimension was omitted). This derived scale was subsequently used in other IS
service quality studies such as Kang and Bradley (2002) and Park and Kim (2005).
The concept of the zone of tolerance (ZOT) (Parasuraman et al., 1994b), was another service-quality
concept that captured the attention of IS researchers. The ZOT is a concept which allows for the multi-level
nature of the user expectation measure. It provides for the measurement of the difference in user
expectation between what the user considers an adequate level and that of a desired level of service. In a
recent paper Kettinger and Lee (2005) again used the IS-adapted 22-item SERVQUAL scale to empirically
test the ZOT concept in the IS domain. This time they derived an 18 Item scale across four dimensions.
Unlike their previous findings (Kettinger and Lee, 1997) the Tangibles dimension was retained, and the
Assurance and Empathy dimensions merged to form a new dimension which the authors labelled Rapport.
Thus the dimensions for the derived 18-item scale, which the authors called IS-ZOT-SERVQUAL, are
Reliability, Responsiveness, Rapport and Tangibles. However, both the Kettinger and Lee (1997, 2005)
studies were conducted in the context of large brick-and-mortar organisations.
The foregoing review supports the assertions in the introduction of this paper regarding the dominance of IS
service quality studies within large organisations only. In the next section, the research design and methods
are discussed. In particular, we describe the use of a survey design, the delineation of the research
population, and the statistical procedures that were applied to analyse the data.
3. Research design and methodology

Following on studies such as Kang and Bradley (2002), Kettinger and Lee (1997, 2005), Pitt et al. (1995) and
Watson et al. (1998), the empirical work for this study was conducted using survey design principles.
The main section of the survey questionnaire consisted of the IS-adapted SERVQUAL questions that
Kettinger and Lee used in their study (Kettinger & Lee, 2005). Kettinger and Lees adaptation entailed
rewording of the 22 items 3 to reflect the IS environment. These scale items were then reformatted for the
current research, to focus only on the expectations of the clients of IS service providers i.e. e-Commerce
Cronin and Taylor (1994) critiqued the use of the SERVQUAL perception-minus-expectations (P-E) measure in
favour of a perception-only measure. They called their perception-only measuring instrument SERVPERF.
Parasuraman et al. (1994a) refuted many of these concerns, and defended the managerial diagnostic capability of
SERVQUAL over SERVPERF.
3
In their 2005 study, Kettinger and Lee tested the 22-item SERVQUAL scale by adapting the wording for the IS
environment. We have used this IS-adapted 22-item scale in this study.
www.ejise.com
112
SME managers. Additionally an open-ended question was included in the survey instrument to draw possible
inferences about any additional service quality expectations in the research context.
3.1 Research population

The first consideration in choosing the research population was that homogeneous data from the same
sector was needed, so that data from different respondents would be comparable. It would thus also be
unproblematic to aggregate the data for statistical analysis.
The tourism sector was selected because of the vast number of SMEs in this sector in South Africa (Warden
& Williams, 2003). The tourism sector was also well suited to the application of e-Commerce and was an
early adopter of e-Commerce in this country (Wynne & Berthon, 2001).
As this research focused on the service requirements of e-Commerce enabled businesses, the research
population chosen for this study, from within the tourism sector, was e-Commerce enabled bed-andbreakfast and self-catering accommodation businesses in the Western Cape 4 . An e-Commerce enabled
business, for the purposes of this study, was defined as a business that was on an adoption level of 2 or
higher on the Subba Rao and Metts (2003) e-Commerce adoption stage model (refer to Table 2).
Table 2: Subba Rao and Metts (2003) e-Commerce adoption stage model levels
Adoption Stage Level
1. Presence
2. Portals
3. Transactions Integration
4. Enterprises Integration
Stage Characteristics
Web Content; Window to the Web; No Integration; E-mail
Profiles; two-way Communication; E-mail; Order Placing; Cookies; No
payment Transactions
B2B/B2C; Communities; E-Marketplaces; Auctions; 3rd Party EMarketplaces; Low level Collaboration; Payment Transactions
E2E; Full Integration; E-Business; E-commerce + CRM + SCM; Value
Chain Integration; High level Collaboration
In summary, the criteria for eligibility to participate in the survey were that the business had to be:
A bed-and-breakfast or self-catering accommodation business;
Based in the Western Cape, South Africa;
An SME; and
At a minimum level 2 on the adoption model of Subba Rao and Metts (2003).
3.2 The survey

The Capestay website (http://www.capestay.co.za) which is a comprehensive directory for all types of
accommodation in the Western Cape was used as a basis for random sampling. This website featured
category listings which allowed for the separate listings of the bed-and-breakfast and self-catering
businesses. The website hosts a dedicated homepage for each listing, which provides detailed information
about the establishment, as well as an online booking facility. This implied that the businesses listed would
be on a minimum of Level 2 (Portals Stage) on the Subba Rao and Metts (2003) e-Commerce adoption
stage model. The listings on this directory website thus provided a representative list from which to select a
sample for the research.
The www.capestay.co.za directory website had 1177 unique contact email addresses for bed-and-breakfast
and self-catering accommodation establishments based in a wide geographic spectrum of the Western Cape
region. The survey questionnaire accompanied by a cover letter was emailed to the listed 1177 email
addresses. Only 48 responses were received. This is a response rate of only 4%, which although was lower
than expected 5 , is a trend corroborated by Sheehan (2001) for email surveys.
3.3 Methodology
The survey instrument, based on the IS-adapted SERVQUAL instrument, used the same dimensions and
items which had been applied in the context of large brick-and-mortar businesses in other studies. The
validity of these dimensions and respective items for the research context (e-Commerce SMEs) was
statistically tested using Exploratory Factor Analysis and re-affirmed using Confirmatory Factor Analysis.
4
The Western Cape is one of nine provinces in the Republic of South Africa.
Previous experience with the micro business sector in this region indicated that owners and managers were generally
not very responsive to surveys.
www.ejise.com
113
ISSN 1566-6379
The survey respondents were asked to rate the importance of individual service quality items taking into
account their relationship with IS service providers. The objective was to assess the respondents service
quality expectations. The mean values of all the item responses per dimension were then used to rank the
service quality dimensions. Thereafter EFA was used to statistically conclude if any service quality items
should be eliminated and/or if the service dimensionality needed to be adapted for the research context.
4. Data analysis
The statistical data analysis for this research followed an approach similar to the approach used by Kettinger
and Lee (2005). The analysis was undertaken as follows:
1. Perform an Exploratory Factor Analysis (EFA), using Principal Component Analysis, to ascertain if there
are new or different service quality dimensions in this research context. This essentially entails eliminating
low-scoring instrument items and regrouping others.
2. Perform a Confirmatory Factor Analysis (CFA) on the adapted survey instrument to re-affirm the results
of the EFA.
3. Assess the validity and reliability of the adapted instrument, using Cronbach alpha and the results of the
CFA.
The results of the foregoing are reported on in the following sections.
4.1 Exploratory factor analysis

EFA using a Principal Component Analysis (PCA) extraction method, is used to derive the minimum number
of factors that account for the maximum portion of the total variance in an exploratory manner (Kettinger &
Lee, 2005:612). An exploratory approach, similar to the approach used by Kettinger and Lee (2005), was
also used in the current research. The following factor selection criteria was used in the EFA:
1. The use of Oblique rotation i.e. using Oblimin Rotation Method;
2. Factor loading should be greater than or equal to 0.5;
3. No multiple loadings are allowed i.e. no items (rows) with multiple factor loadings greater or equal to 0.5;
4. No single loadings i.e. no factors (columns) having only one high loading item.
Items not fulfilling the selection criteria were eliminated, as indicated by a strikethrough line in Table 3. A
bold font with shading indicates the highest factor loading for each of the remaining items in Table 3.
www.ejise.com
114
Table 3: Exploratory factor analysis

Component
Relia1 provides you with services as promised?
Relia2 is dependable in handling your service
problems?
Relia3 performs services right the first time?
.426
.743
-.055
.156
-.453
.028
.639
.141
.175
-.116
.223
.500
.194
.304
-.732
Relia4 provides you with services at the promised

time?
.280
.582
.028
.656
-.616
Relia5 provides you with reliable technology and

systems?
.276
.161
.229
.647
-.134
.377
.136
.224
.182
-.719
Resp1 keeps you informed about when service

will be made?
Resp2 delivers prompt service to you?
.312
.037
.290
.123
-.838
Resp3 has the willingness to help you?
.429
.661
.137
.397
-.595
Resp4 has the readiness to respond to your

requests?
.796
.134
.273
.290
-.342
A1 has staff that instils confidence in you?
.870
.112
.319
.650
-.222
A2 makes you feel safer in computer

transactions?
.673
-.032
.301
.728
-.479
A3 has staff that is consistently courteous?
.842
.148
.452
.440
-.366
A4 has staff that has the knowledge to answer

your questions?
.549
.674
.155
.349
-.164
.882
.179
.191
.235
-.200
.873
.043
.406
.458
-.421
E1 gives you individual attention?

E2 has staff that deal with you in a caring
fashion?
E3 has your best interest at heart?
.719
-.278
.288
.640
-.261
E4 has staff that understands your service

needs?
.286
.115
.154
.912
-.220
E5 has convenient business hours?
.039
.053
.216
.253
-.650
T1 has up to date technology?

T2 has visually appealing premises and facilities?
.426
-.336
.248
.312
-.327
.220
.022
.894
.116
-.237
T3 has staff who appear professional?
.426
-.036
.832
.192
-.150
T4 has useful support materials (such as

documentation, training, videos, etc.)?
.140
.151
.794
.415
-.471
Extraction Method: Principal Component Analysis.

Rotation Method: Oblimin with Kaiser Normalization.
4.2 Regrouping of instrument items

As indicated in Table 3, eight items were eliminated, with all five dimensions being affected. After regrouping
the remaining 14 items according to the highest factor loading, only one of the original SERVQUAL
dimensions emerged from the Principal Component Analysis i.e. Tangibles as Component 3 in Table 3.
Components 1, 4 and 5 had a mixture of items while Component 2 had only two of the original Reliability
items. Table 4 summarises these item groupings
www.ejise.com
115
ISSN 1566-6379
Table 4: Item regrouping

COMPONENT 1 = Supportiveness
Resp4 has the readiness to respond to your requests?
E2 has staff that deal with you in a caring fashion?
COMPONENT 2 = Credibility
Relia2 is dependable in handling your service problems?
COMPONENT 3 = Tangibles
T4 has useful support materials (such as documentation, training, videos, etc.)?
COMPONENT 4 = Expertise
Relia5 provides you with reliable technology and systems?
E4 has staff that understands your service needs?
COMPONENT 5 = Availability
Resp1 keeps you informed about when service will be made?
The derived dimensions shown in Table 4 contained a mixture of items from the original set of five
dimensions in the IS-adapted SERVQUAL scale. These derived dimensions were then relabelled. The labels
were intuitively chosen based on the criteria that the grouped items represented, i.e. each group of data was
labelled based on the imagery of meaning they evoked when examined comparatively and in context (See
Strauss & Corbin, 1998:105 for a more detailed explanation of labelling or coding textual data). 6 The
research by Wilkin and Castleman (2003) also helped guide the formulation of appropriate labels for the
derived dimensions, viz., Expertise, Credibility, Availability, and Supportiveness. The rationale for these
labels are as follows:
Component 1 had a mixture of Responsiveness, Assurance and Empathy items. These items related to the
supportive interaction and communication between the service provider and its customer. An appropriate
dimension label was Supportiveness to frame the supportive service quality aspects of the service provider
towards its customers.
Component 2 contained only items from the original Reliability dimension. However, one other Reliability
item had moved to Component 4. The remaining two items dealt with issues of service provider credibility
regarding keeping service promises and handling service problems. An appropriate dimension label to frame
these items was Credibility.
Component 3 contained only items from the original Tangibles dimension. Although one of the original
Tangibles items was omitted, the remaining items were still best framed by the Tangibles dimension label,
which focused on appearances and support materials. Thus this dimension label was retained.
Component 4 had a mixture of Reliability and Empathy items. These items related to the capability of the
service provider to deliver reliable technology and systems that fulfil the customer needs. An appropriate
dimension label for these items was Expertise, framing the ability of the service provider to provide the
required systems.
Component 5 had a mixture of Responsiveness and Empathy items. These items related to the timely
delivery of service. An appropriate dimension label was Availability to frame issues dealing with duration
and promptness of service.
6
A similar approach was followed by Kettinger and Lee (2005). One of the derived dimensions in their research had a
mixture of Empathy and Assurance items. This derived dimension was then relabelled Rapport by the authors because
the construct items focus on an IS service providers ability to convey a rapport of knowledgeable, caring, and courteous
support (Kettinger & Lee, 2005:612).
www.ejise.com
116
4.3 Confirmatory factor analysis (CFA)

The next step in the data analysis was to perform a CFA on the data. A CFA using a Principal Axis Factoring
extraction method was performed on the newly derived 14-item instrument. This was performed to confirm
the dimensionality of this instrument (refer to Table 4). However, unlike Kettinger and Lee (2005), who used
a second set of sample data (holdout sample) to confirm the dimensionality, this research used the same set
of collected research data since the total number of usable responses were relatively small. The process
followed in this research was therefore not as refined as the approach used by Kettinger and Lee (2005) and
is noted as a limitation. The results of the CFA are presented in Table 5.
Table 5: Confirmatory factor analysis
1
.797
.291
.138
-.272
.300
.871
.456
.095
-.413
.486
.826
.176
.150
-.303
.248
.839
.405
.063
-.512
.491
.240
.808
.038
-.328
.197
.400
.779
-.037
-.176
.237
T4 has useful support materials (such as

documentation, training, videos, etc.)?
.161
.746
.282
-.513
.448
.445
-.032
.469
-.419
.134
.091
.056
.875
-.122
.094
.344
.204
.159
-.835
.203
.331
.314
.102
-.645
.156
.122
.195
.138
-.499
.273
.263
.201
.010
-.192
.485
.247
.159
.144
-.205
.826

EXPERTISE
AVAILABILITY
CREDIBILITY
TANGIBLE
S
SUPPORTIVENESS
Factor

Relia2 is dependable in handling your service
problems?
Resp1 keeps you informed about when service will be

made?
Relia5 provides you with reliable technology and

systems?
Extraction Method: Principal Axis Factoring.

Rotation Method: Oblimin with Kaiser Normalization.
Garson (2007) defines a rule-of-thumb for factor loadings values to be weak if the value is less than 0.4,
strong if it is more than 0.6, otherwise it is moderate. Applying this rule-of-thumb to Table 5, three of the
factor loadings are considered moderate and 11 are considered strong. The results of the Confirmatory
Factor Analysis thus appear to confirm the derived dimensions of the EFA in Table 3.
4.4 Validity and reliability of the derived instrument

Instrument validity entails verifying that the constructs measured by the instrument are real and reliable, and
that the instrument is measuring the right content (Straub et al., 2004). Content validity is defined by Straub
et al. (2004) as the indication of whether the instrument is a true representation of all the ways that could be
used to measure the content of the given construct. These authors assert that content validity is established
www.ejise.com
117
ISSN 1566-6379
through literature reviews and expert judges or panels. The items used in this survey instrument were based
on SERVQUAL items originally tested for applicability in the IS domain by Pitt et al. (1995). Therefore the
rationalization used in Pitt et al. (1995) to argue the Content Validity of the measuring instrument also applies
to this research.
Straub et al. (2004) posit that the two main components of Construct Validity, viz., convergent Validity and
discriminant validity, can be deduced from the CFA results. The strong factor loadings indicate good
convergent validity, because the items converge strongly to the derived dimensions. Also discriminant
validity can be deduced because the factor loadings indicate that the items do not overlap across different
dimensions.
Reliability is an evaluation of the measurement of accuracy of the instrument and can be viewed as the
extent to which the respondent can answer the same questions or close approximations the same way each
time (Straub, 1989).
A Cronbach alpha measurement can be used to determine reliability of a measurement instrument (Straub,
1989). A Cronbach alpha measurement of 0.7 and greater is considered reliable (Straub et al. 2004). The
Cronbach alpha measurement for the derived instrument was calculated at 0.837 thus indicating good
reliability.
4.5 Relative importance of the derived dimensions

The mean of the survey responses (rated 1 to 5, with 5 being most important) for each survey item, was
used to deduce the importance ranking of the specific item i.e. the item with the highest mean value was
regarded as the most important. The average score for each item of the derived factors (see Table 4) are
tabulated in Table 6. The mean of all items that comprised each of the factors was then calculated. These
mean values are indicated by a bold shaded font in Table 6.
Table 6: Mean values for regrouped items
FACTOR
Mean per
item
Mean for
Factor
FACTOR 1 = Supportiveness
4.69
4.00
4.35
3.96
FACTOR 2 = Credibility
Relia2 is dependable in handling your service problems?
4.92
4.79
FACTOR 3 = Tangibles
2.15
3.29
T4 has useful support materials (such as documentation, training, videos, etc.)?
3.04
FACTOR 4 = Expertise
Relia5 provides you with reliable technology and systems?
4.81
4.67
4.25
4.86
2.83
4.74
FACTOR 5 = Availability
Resp1 keeps you informed about when service will be made?
4.56
4.75
4.42
4.58
The results in Table 6 were used to deduce the relative importance of the derived dimensions in the research
context. The ranking of the service quality dimensions in the research context was as follows:
1st Credibility
www.ejise.com
118

2nd Expertise
3rd Availability
4th Supportiveness
5th Tangibles
5. Analysis of responses to the open-ended question

In this the last part of the data analysis section, the responses to the open-end question in the survey
instrument are analysed i.e.
How can your e-Commerce service providers better assist or support your business to benefit from eCommerce?
The verbatim responses are included in the Appendix.
In taking an interpretive approach (Klein & Myers, 1999) the text was analysed for both literal and underlying
meaning. However, no conclusions from the responses could be made on whether additional service quality
dimensions are applicable to the research context. However, the responses seem to support the derived
dimensions of service quality in the e-Commerce SME context. The impacted dimensions for each of the
verbatim responses are also included in the Appendix. A summary of the results of the number of responses
impacting on these dimensions are displayed in Table 7.
Table 7: Service Quality (SQ) dimension impacts of the responses to open-ended question
SQ Dimension
Credibility
Expertise
Availability
Supportiveness
Tangibles
No. of responses impacting on the SQ dimensions

11
11
5
4
2
The results in Table 7 thus appear to confirm the results of the importance ranking of the derived dimensions
in the previous section.
6. Data interpretation
The EFA results indicate that all the SERVQUAL items are not needed in the research context i.e. bed-andbreakfast and self-catering accommodation eCommerce SME's in the tourism sector. Eight items were
consequently omitted. These items are not necessarily unimportant, but the results suggest that these are
not required in the measurement of service quality expectations in the business environment studied. The
remaining items are sufficient for gathering data about the relative importance of the derived service quality
dimensions. The derived dimensions, viz., Credibility, Expertise, Availability, Supportiveness and Tangibles,
indicate the expected service quality focus in the research context.
An explanation of the meaning and attributes of these dimensions are tabulated in Table 8.
Table 8: Meaning and Attributes of the Derived Service Quality Dimensions
Derived
Dimension
Credibility
Expertise
Availability
Supportiveness
Tangibles
www.ejise.com
Meaning and Attributes of the Derived Dimension

The service provider should be credible in maintaining its service promises and delivering the exact
system requirements as agreed with the client. The service provider should also be dependable when
handling service problems after system installation.
The service provider should have the expertise to deliver reliable systems and know-how to its clients.
The service provider should also have the ability to understand the service needs of the client, and how
to fulfil those needs.
The service provider should be available when service is required, and should respond promptly to
service requests. The service provider should also be able to inform the client about time, duration and
status of service requests.
The service provider should have the readiness to help clients in a caring and supportive manner.
The service provider should have visually appealing premises, staff that appear professional, and supply
useful support materials.
119
ISSN 1566-6379
Figure 3 illustrates the change of service quality dimensionality for the specific business environment
studied. The Tangibles dimension label was retained from SERVQUAL, although one of its original items
was omitted. The remaining items were still best framed by this label. The other SERVQUAL dimension
labels, viz., Reliability, Responsiveness, Assurance and Empathy, were replaced by Credibility, Availability,
Expertise and Supportiveness.
Figure 3: Change of service quality dimensions for research context
7. Conclusion
The objective of this study was to investigate the application of an IS-adapted SERVQUAL instrument in the
SME e-Commerce sector. Evidence has been provided that in the SME sector chosen for this study, service
quality expectations differ from that of larger corporate organisational settings. The results of the study have
yielded a validated instrument with a new set of service-quality dimensions. The dimension differences
between the e-Commerce and brick-and-mortar contexts could be due to issues of trust and uncertainty in
an environment that relies on outsourced IS service providers. Additionally issues related to the separation of
human contact and required technology expertise could also have an influence. These issues are expanded
on in the framing of each of the new service quality dimensions that follows.
Credibility was identified as being the most important service quality dimension. The criteria for this
dimension entail that the IS service provider be credible in maintaining its service promises and delivering
the exact system requirements as agreed with the client. The service provider should also be dependable
when handling service problems after system installation. This could be as a result of the technical
dependence that the SMEs have on their IS service providers. The high ranking of this dimension could also
be related to uncertainty of the security and reliability of the online environment, especially after the dotcom crash (Razi et al., 2004). Thus IS service providers need to instil confidence in their e-Commerce SME
clients. One of the implications of this is that they should refrain from making service promises that they
know will be difficult to honour. This also has a bearing on the extent to which SMEs successfully adopt eCommerce, since they can only be expected to embrace the technology if service providers are perceived to
be credible.
Expertise was identified as the second most important service quality dimension. The criteria for this
dimension suggest that the IS service provider is expected to have the expertise to deliver reliable systems
and technical know-how to its clients. This dimension also incorporates the service providers ability to
understand the service needs of the client, and how to go about fulfilling those needs, and is possibly related
to the SMEs technical dependence on IS service providers. Barnes et al. (2004) report that outsourcing IS
expertise was one of the problems during the dot-com crash. They assert that this was a weakness in eCommerce businesses with respect to initial technology choices and on-going management and
development. The lack of technical expertise by the SMEs could place additional requirements on the
service provider in solving service problems. The service provider would need to have the expertise to also
get to the root cause of service problems without technical advice from the client. It is very important that the
IS service providers consistently provide a high level of expertise, and in so doing allow the e-Commerce
SMEs to concentrate on their core business activities.
The third ranked service quality dimension is Availability. The criteria for this dimension suggest that the IS
service provider be available when service is required, and respond promptly to service requests. The
service provider should also be able to inform the client about time, duration and status of service requests.
The online environment places large importance on availability of Internet technologies. System downtime
could mean revenue loss for the e-Commerce SME. These businesses firstly require reliable systems that
seldom fail. But if the systems do fail, they expect prompt service and reparations. The businesses expect
www.ejise.com
120
the service providers to be available when there are system failures, and to have the processes and
infrastructure in place to keep them updated on reparation progress.
The fourth ranked service quality dimension is Supportiveness. The criteria for this dimension underscore the
willingness of the IS service provider to assist clients with a caring and supportive disposition. This
dimension underscores the need for service providers to be able to empathise with the IS related problems
that underlie business problems confronting managers. IS service providers are evaluated by clients based
on mutual interests, shared approaches to problem solving, and a compatible management culture and style
(Dibbern et al., 2004). This is the foundation on which the service relationship is structured. It is thus
important for IS service providers to provide such supportiveness in order to foster goodwill and trust in their
clients.
The last dimension, and lowest in terms of importance is Tangibles. The criteria for this dimension suggest
that the IS service provider have visually appealing premises, staff that appear professional, and supply
useful support materials. The finding that Tangibles is the least important dimension in the research context
is not surprising considering the nature of an outsourcing relationship. SME businesses in the e-Commerce
environment are less dependant on visual contact with the service provider. IS service providers in this
environment are able to manage technical and other problems remotely. Consequently less physical
interaction required. Thus IS service providers should focus on the delivery of reliable systems rather than on
attractive and professional visual appearances of staff and premises.
Lastly an understanding of these service quality dimensions is not only useful to the IS service providers, but
to the e-Commerce SME business managers themselves. Business managers with a higher level of prior
experience, and greater familiarity with the subject of evaluation may be more confident about the realisation
of their expectations (Khalifa & Liu, 2003). These dimensions and their criteria serve to highlight to the
business managers what reasonable service quality expectations are in this business environment.
The study provides the foundation for much needed research in the SME sector to enhance the ability of
businesses to conduct e-Commerce business on a satisfactory platform of IS. The study highlights those
areas in which SMEs require support from their IS service providers. Future work will entail further testing of
the derived instrument in other SME sectors, as well as application of the instrument amongst clients of
specific IS service providers.
Acknowledgements
The authors gratefully acknowledge:
Financial support from the National Research Foundation (NRF) of South Africa, and of the Cape
Peninsula University of Technology (CPUT) for the financial contribution to this study.
N.B. Any opinion, findings and conclusions or recommendations expressed in this paper, are those of the
authors and do not necessarily reflect the views of the NRF.
Corrie Uys, e-Innovation Academy (CPUT) for statistical support.
References
Al-Qirim, N.A.Y & Bathula, H. 2002. IT and eCommerce Outsourcing in Small to Medium-Size Enterprises in New
Zealand: An Exploratory Research. Proceedings of the 15th Bled Electronic Commerce Conference eReality:
Constructing the eEconomy, Bled, Slovenia, 17-19 June 2002.
Barnes, D., Hinton, M. & Mieczkowska, S. 2004. Avoiding the fate of the dotbombs: lessons from three surviving dotcom
start-ups. Journal of Small Business and Enterprise Development, 11(3):329-337.
Bharati, P. & Berg, D. 2003. Managing information systems for service quality: a study from the other side. Information
Technology & People, 16(2):183-202.
Cronin, J.J., Jr. & Taylor, S.A. 1994. SERVPERF versus SERVQUAL: reconciling performance-based and perceptionminus-expectations measurement of service quality. Journal of Marketing, 58(1):125-131.
DeLone, W.H. & McLean, E.R. 1992. Information systems success: the quest for the dependant variable. Information
Systems Research, 3(1):60-95.
DeLone, W.H. & McLean, E.R. 2003. The DeLone and McLean model of information systems success: a ten-year
update. Journal of Management Information Systems, 19(4):9-30.
Dibbern, J., Goles, T., Hirschheim, R. & Jayatilaka, B. 2004. Information systems outsourcing: a survey and analysis of
the literature. The DATA BASE for Advances in Information Systems, 35(4):6-102.
Elpez, I. & Fink, D. 2006. Information systems success in the public sector: stakeholders perspectives and emerging
alignment model. Issues in Informing Science and Information Technology, 3:219-231.
Garson, G.D. 2007. Factor analysis. [online] http://www2.chass.ncsu.edu/garson/pa765/factor.htm [14 March 2008].
www.ejise.com
121
ISSN 1566-6379
Grover, V., Jeong, S.R. & Segars, A.H. 1996. Information systems effectiveness: The construct space and patterns of
application. Information & Management, 31:177-191.
Jiang, J.J., Klein, G. & Carr, C.L. 2002. Measuring information system service quality: SERVQUAL from the other side.
MIS Quarterly, 26(2):145-166.
Kang, H. & Bradley, G. 2002. Measuring the performance of IT services: an assessment of SERVQUAL. International
Journal of Accounting Information Systems, 3(1):151-164.
Kettinger, W.J. & Lee, C.C. 1997. Pragmatic perspectives on the measuring of information systems service quality. MIS
Quarterly, 21(2):223-241.
Kettinger, W.J. & Lee, C.C. 2005. Zones of tolerance: alternative scales for measuring information systems service
quality. MIS Quarterly, 29(4):607-623.
Khalifa, M. & Liu, V. 2003. Determinants of Satisfaction at Different Adoption Stages of Internet-Based Services. Journal
of the Association for Information Systems, 4(5):206-232.
Kim, D.R., Cheon, M.J., Beugre, C.D. & Coverdale, T.K. 2003. Information systems outsourcing: exploration on the
impact of outsourcing service providers service quality. Issues in Information Systems (IIS), 4(2):528-534.
Kim, K.K. 1990. User Information Satisfaction: Toward Conceptual Clarity. Proceedings of the International Conference
on Information Systems (ICIS), Copenhagen.
Klein, H. K. & Myers, M. 1999. A set of principles for conducting and evaluating interpretive field studies in information
systems. MIS Quarterly, 23(1):67-94.
Lomerson, W.L. & Tuten, P.M. 2005. Examining evaluation across the IT value chain. Proceedings of the 2005 Southern
Association of Information Systems Conference, Savannah, Georgia, USA, 2526 February 2005:124-129.
Lui, C. & Arnett, K.P. 2000. Exploring the factors associated with Web site success in the context of electronic
commerce. Information & Management, 38(1):23-33.
Molla, A. & Licker, P. 2001. E-Commerce systems success: an attempt to extend and respecify the DeLone and McLean
model of IS success. Journal of Electronic Commerce Research, 2(4):131-141.
Parasuraman, A., Berry, L.L. & Zeithaml, V.A. 1993. Research note: more on improving quality measurement. Journal of
Retailing, 69(1):140-147.
Parasuraman, A., Berry, L.L. & Zeithaml, V.A. 1994b. Alternative scales for measuring service quality: a comparative
assessment based on psychometric and diagnostic criteria. Journal of Retailing, 70(3):201-230.
Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1985. A conceptual model of service quality and its implications for future
research. Journal of Marketing, 49(4):41-50.
Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1988. SERVQUAL: a multiple-item scale for measuring consumer
perceptions of service quality. Journal of Retailing, 64(1):12-40.
Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1991. Refinement and reassessment of the SERVQUAL scale. Journal of
Retailing, 67(4):420-450.
Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1994a. Reassessment of expectations as a comparison standard in
measuring service quality: implications for further research. Journal of Marketing, 58(1):111-124.
Park, J. & Kim, J.S. 2005. The impact of IS sourcing type on service quality and maintenance efforts. Information &
Management, 42(1):261-274.
Pather, S., Remenyi, D. & Erwin, G. 2004. E-commerce success: the quest for IS effectiveness measurement: a
conceptual framework for the e-commerce environment. South African Computer Journal, 32:34-43.
Pitt, L.F., Watson, R.T. & Kavan, C.B. 1995. Service quality: a measure of information systems effectiveness. MIS
Quarterly, 19(2):173-188.
Pitt, L.F., Watson, R.T. & Kavan, C.B. 1997. Measuring information systems service quality: concerns for a complete
canvas. MIS Quarterly, 21(2):209-223.
Razi, M.A., Tarn, J.M. & Siddiqui, F.A. 2004. Exploring the failure and success of DotComs. Information Management &
Computer Security, 12(3):228-244.
Rohde, F.H. 2004. IS/IT outsourcing practices of small- and medium-sized manufacturers. International Journal of
Accounting Information Systems, 5:429-451.
Seddon, P.B. & Kiew, M-Y. 1996. A Partial Test and Development of DeLone and McLean's Model of IS Success.
Australian Journal of Information Systems, 4(1):90-110.
Sheehan, K. 2001. E-mail survey response rates: a review. Journal of Computer-Mediated Communication, 6(2).
http://jcmc.indiana.edu/vol6/issue2/sheehan.html [14 March 2008].
Stansfield, M. & Grant, K. 2003. An investigation into issues influencing the use of the internet and electronic commerce
among small-medium sized enterprises. Journal of Electronic Commerce Research, 4(1):15-33.
Straub, D., Boudreau, M. & Gefen, D. 2004. Validation guidelines for IS positivist research. Communications of the
Association for Information Systems, 13:380-427.
Straub, D.W. 1989. Validating instruments in MIS research. MIS Quarterly, 13(2):147-169.
Strauss, A.L. & Corbin, J. 1998. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded
Theory, 2nd ed. London: Sage.
Subba Rao, S. & Metts, G. 2003. Electronic commerce development in small and medium sized enterprises: a stage
model and its implications. Business Process Management Journal, 9(1):11-32.
Van Dyke, T.P., Kappelman, L.A. & Prybutok, V.R. 1997. Measuring information systems service quality: concerns on the
use of the SERVQUAL questionnaire. MIS Quarterly, 21(2):195-208.
Warden, S. & Williams, R. 2003. Benefits for small medium and micro enterprises (SMMEs) in the Western Cape in the
th
tourism industry for adopting electronic commerce. Proceedings of the 5 Annual Conference on World Wide Web
Applications, Rand Afrikaans University, 10-12 September 2003.
www.ejise.com
122
Watson, W.T., Pitt, L.F. & Kavan, C.B. 1998. Measuring information systems service quality: lessons from two
longitudinal case studies. MIS Quarterly, 22(1):61-79.
Watson, W.T., Pitt, L.F., Cunningham, C.J. & Nel, D. 1993. User satisfaction and service quality of the IS department:
closing the gaps. Journal of Information Technology, 8:257-265.
Whyte, G. & Bytheway, A. 1996. Factors affecting information systems success. International Journal of Service Industry
Management, 7(1):74-93.
Wilkin, C. & Castleman, T. 2003. Development of an Instrument to Evaluate the Quality of Delivered Information
Systems. Proceedings of the 36th Hawaii International Conference on System Sciences, Big Island, HI, USA, 6-9
January 2003.
Wilkin, C., Carr, R. & Hewett, W. 2001. Evaluating IS quality: exploration of the role of expectations on stakeholders'
evaluation. In Van Grembergen, W. (ed). Information Technology Evaulation: Methods and Management. Hershey
PA: Idea Group Publishing: 111-129.
Wynne, C. & Berthon, P. 2001. The impact of the Internet on the distribution value chain: the case of the South African
tourism industry. International Marketing Review, 18(4):420-431.
Appendix
Verbatim responses to Open-ended Survey Questions
How can your e-Commerce service providers better assist or support your business to benefit from eCommerce?
The verbatim responses to this question are listed below together with the service quality dimensions which
are impacted by the responses:
Dont let the server go down loss of business SP should do what they say they can and will do.
IMPACTED DIMENSIONS: Credibility, Expertise
SP should care about me I should be important enough to them they should know who I am. Had a
bad experience with Webmail did not deliver on promise wasted R6000.
IMPACTED DIMENSIONS: Credibility, Supportiveness
SP should deliver on promises of increased customer awareness and bookings. Have listings on 23
website get booking from only 3. Have invested R30 000 in listings.
IMPACTED DIMENSIONS: Credibility

User-friendly templates.
IMPACTED DIMENSIONS: Tangibles

Keeping us informed of technological advances that may be of benefit.
IMPACTED DIMENSIONS: Expertise, Availability

They could give themselves a face! We have never met most of them - purely email or phone
conversations.
IMPACTED DIMENSIONS: Supportiveness, Tangibles
By constantly reviewing my website and ensuring that the website appears at the top of search engine
listings.
IMPACTED DIMENSIONS: Availability, Expertise
Our e-Commerce service provider does keep as well abreast of new technologies, systems and
developments. We feel secure in their hands, and rely heavily on their expertise going forward. They have a
good understanding of our business model. This we feel is the most important part of an e-Commerce
provider, that they take the time to understand and develop with you (not for you) an e-Commerce solution to
suit your business.
I rely on good websites with loads of hits leading to loads of enquiries. All my business comes from
websites. Most I try are a waste of money but a couple of South African ones always come up trumps. I now
don't subscribe to any new offers unless they give a free trial and can put their money where their mouth is.
Note: I am with MWEB who have consistently provided the best service around at almost the best
rates.
Probably more than 90% of our business is sourced from our secondary website listings. I think the
small B&B market is well catered for in SA. I'm generally very satisfied with the services provided.
www.ejise.com
123
ISSN 1566-6379
More information for latest developments and current improvements. I am happy with my service
provider right now but need to sort out Telkom and the skyrocketing costs of dial-up. Honestly, these are
necessary evils.
Make sure that the business advertised reach more possible clients. Help me in my marketing endevour
and win customers for my tourist business.
IMPACTED DIMENSIONS: Expertise
I find my dealings with the e-Commerce service providers outstanding and personal attention
exceptional!
IMPACTED DIMENSIONS: Availability, Supportiveness

Dont interrupt service delivery.
IMPACTED DIMENSIONS: Availability

They close over December January, but I suppose they need to take a break.
IMPACTED DIMENSIONS: Availability

Up to date with technology and developments in that area.

Update and Upgrade website. Confidentiality. Improve search engine ratings.

SP should inform me of webpages that are not working for me are they viable.

None, very happy with the service that I receive, good feedback & reasonable special offer advertising
capestay.co.za
IMPACTED DIMENSIONS: Credibility, Supportiveness

By taking a pro-active approach.

Ensure that search engines find our website.
www.ejise.com
124
Use and Impact of ICT on SMEs in Oman

Rafi Ashrafi and Muhammed Murtaza
Sultan Qaboos University, Oman
rafi@squ.edu.om
muhd.murtaza@gmail.com
Abstract: This paper presents the results of an exploratory study carried out to learn about the use and impact of
Information and Communication Technologies (ICT) on Small and Medium Sized Enterprises (SMEs) in Oman. The
study investigates ICT infrastructure, software used, driver for ICT investment, perceptions about business benefits of
ICT and outsourcing trends of SMEs. The study provides an insight on the barriers for the adoption of ICT. Data on these
aspects of ICT was collected from 51 SMEs through a survey instrument. The results of the study show that only a small
number of SMEs in Oman are aware of the benefits of ICT adoption. The main driving forces for ICT investment are to
provide better and faster customer service and to stay ahead of the competition. A majority of surveyed SMEs have
reported a positive performance and other benefits by utilizing ICT in their businesses. Majority of SMEs outsource most
of their ICT activities. Lack of internal capabilities, high cost of ICT and lack of information about suitable ICT solutions
and implementation were some of the major barriers in adopting ICT. These findings are consistent with other studies
e.g. (Harindranath et al 2008). There is a need for more focus and concerted efforts on increasing awareness among
SMEs on the benefits of ICT adoption. The results of the study recognize the need for more training facilities in ICT for
SMEs, measures to provide ICT products and services at an affordable cost, and availability of free professional advice
and consulting at reasonable cost to SMEs. Our findings therefore have important implication for policy aimed at ICT
adoption and use by SMEs. The findings of this research will provide a foundation for future research and will help policy
makers in understanding the current state of affairs of the usage and impact of ICT on SMEs in Oman.
Keywords: Information and communication technologies (ICT), Small and Medium Sized Enterprises (SMEs),
developing countries, Gulf Cooperative Council (GCC), Middle East, Oman
1. Introduction
Today organizations of all types are utilizing Information and Communication Technologies (ICT) around the
globe, not only for cutting costs and improving efficiency, but also for providing better customer service.
Governments too, around the world, are adopting ICT to provide better services to their citizens. The
adoption of ICT by organizations requires a business environment encouraging open competition, trust and
security, interoperability and standardization and the availability of finance for ICT (UNCTAD 2004).
Most of the large and international organizations in Oman have effective computer systems to efficiently
conduct business. A number of large organizations have spent huge amounts of money on installing
computer systems to support their business processes. However, the situation has not been the same with
SMEs - similar to other parts of the world for various reasons (Parker and Castelman 2007), (Shiels et al
2003), and (Fink and Disterer 2006).
No studies have been carried out on this topic in any of the GCC countries. So why Oman? Authors believe
that Oman is a country with unique features such as Oman is a developing country, an oil producing country
but is neither as poor as some of the other developing countries nor as rich as other oil producing countries.
It does not intensely depend on foreign labor as some of the other GCC countries and is expanding its small
industrial base. With its small population and a large geographical area Oman has international standard
grade roads to link all major cities of the country. Similarly education and health facilities are expanded to
rural and remote areas. Telephone, land lines or mobile, and Internet services are available to more than
80% of the population. The female workforce represents 17% of the total workforce of Oman which is a very
distinctive feature as compared to other Arab countries of the region. Oman has limited resources (monetary
and human) and has to use these resources in an efficient way learning from the experiences of other
countries. These and many other geo-political, social and other factors make Oman an interesting case to
study in what aspects Oman is different or similar to other countries in ICT adoption for SMEs.
The government of Oman has taken various measures to diversify the economy for sustainable development
of the country and one of the major steps is to transform Oman into a digital society. The adoption of ICT will
have significant positive consequences on SMEs and consequently on the economy of Oman. There is a
dearth of data and research about the size and contribution of SMEs towards Omans economy. Therefore,
through this research, we would like to learn about the effects and usages of ICT on SMEs in Oman and
their current and future perceptions towards ICT. This study is based on the review of literature on the topic
and the Net Impact Canada 2006 study (Illuminas 2006). We hope that the findings of this research will
ISSN 1566-6379
125
Ashrafi, R. and Murtaza, M. Use and Impact of ICT on SMEs in Oman. The Electronic Journal Information Systems
Evaluation Volume 11 Issue 3 2008, pp. 125 - 138, available online at www.ejise.com
provide a foundation for future research and will help policy makers in understanding the current state of
affairs of the usage and impact of ICT on SMEs in Oman. We collected data on the usage and impact of ICT
on Omani SMEs through a survey questionnaire of 51 randomly selected SMEs in Muscat, the Capital of
Oman.
Definitions: It is important to note that the term "ICT" in context of this research refers to the wide range of
computerized information and communication technologies. These technologies include products and
services such as desktop computers, laptops, handheld devices, wired or wireless intranet, business
productivity software such as text editor and spreadsheet, enterprise software, data storage and security,
network security and others.
SME: There is no official definition of SMEs available. In this research, we have adopted the following
definitions of SMEs. Businesses with less than ten employees as a Micro Enterprise, between ten and fifty
as Small Enterprises, and between fifty to two hundred and fifty employees as Medium sized enterprises.
This criterion is consistent with other similar studies (Kapurubandara et al 2006) and was used to identify
and qualify SMEs for the purpose of this research.
In the next section we review selected literature on the areas related to the topic, followed by diffusion of ICT
in GCC countries and Oman, objectives of the study and research design of the study. In the later sections
we present findings of our study with conclusions, limitations and directions for future research. An earlier
version of this paper was presented at and included in the proceedings of the 2008 International Conference
on Information Resource Management (Conf-IRM) held at Niagara Falls, Ontario, Canada during May 18-20,
2008.
2. Literature review
Importance of SMEs: It has been widely recognized that small and medium enterprises (SMEs) not only
play an important role in the economy of a country, but are crucial to the countrys economic stability. In New
Zealand SMEs make up more than 99% of all businesses and account for about 60% of employment. In the
USA more than half of all the employment comes from firms with fewer than 500 employees (Baldwin et al
2001). In the UK, SMEs employ 67 % of the workforce (Lange et al 2000). In most EU member states SMEs
make up over 99% of enterprises, 67% of jobs and 59% of GDP. In most countries SMEs generate a
substantial share of GDP and a key source of new jobs as well as a breeding ground for entrepreneurship
and new business ideas. The United States of America, UK, Japan, Australia, New Zealand, Canada and
other developed, as well as developing, countries are making policies to facilitate the growth of SMEs.
Realizing the importance of ICT New Zealand spend about 10% of her GDP on ICT, making it the top
ranking country in the world (Clarke 2004). Estimates from the World Bank indicate that SMEs have
contributed over 55% percent of GDP in OECD countries and between 60 to 70 percent of GDP in middleincome and low income countries generating 60 to 70 percent employment (Oman Economic Review 2007).
The above facts show that SMEs play a very important role in the growth of economy of a country, and
Oman is not an exception.
ICT Diffusion in SMEs: There are a number of studies that discuss adoption of Internet and e-business in
SMEs in developed countries (Lucchetti and Sterlacchini 2004), (Love et al 2004), (Schubert and Leimstoll
2006 and 2007a, b), (Koellinger 2006), (Stroeken 2001), (Morikawa 2004), (Caldeira and Ward 2002),
(Gregor et al 2004), and (Doczi 2002). Governments around the globe recognize the importance of adoption
of ICT by SMEs and they have created special groups to study various aspects of ICT adoption in SMEs.
Despite the importance of ICT and emphasis by various governments to encourage SMEs to adopt ICT, it
has been reported that SMEs have been slow in adopting ICT for various reasons (Houghton and Winklhofer
2004), (Smallbone et al 2001), (Dawn et al 2002), and (Lawson et al 2003). We wanted to find out reasons
for the slow adoption of ICT in Oman.
Barriers to ICT Adoption: Large organizations have enough resources to adopt ICT while on the other
hand SMEs have limited financial and human resources to adopt ICT. (Duan et al 2002) identified lack of ICT
skills and knowledge in SMEs as one of the major challenges faced by all European countries, particularly in
the UK, Poland and Portugal, in their study. (Houghton and Winklhofer 2004) have reported a slow response
of SMEs relating to adoption of ICT. (Shiels et al 2003) found that characteristics of the firm and industry
sector are contributory factors to the adoption and exploitation of ICTs by SMEs. (Kapurubandara et al 2006)
have categorized internal and external barriers that impede adoption of ICT by SMEs in a developing
www.ejise.com
126
country. The internal barriers include owner manager characteristics, firm characteristics, cost and return on
investment, and external barriers include: infrastructure, social, cultural, political, legal and regulatory.
ICT Diffusion in developing countries: There are very few studies about ICT adoption in developing
countries (Temtime et al 2003), (Mutula et al 2006), (Yeh et al 2007), (Ssewanyana et al 2007),
(Kapurubandara et al 2006). (Lal 2007) investigating adoption of ICT in Nigerian SMEs, found that one of the
major factors inhibiting ICT diffusion and intensive utilization is poor physical infrastructure. In developing
countries some of the ICT adoption challenges include legal and regulatory issues, weak ICT strategies, lack
of R& D, excessive reliance on foreign technology and ongoing weaknesses in ICT implementation (Dutta et
al 2003).
3. ICT diffusion in Gulf Cooperation Council (GCC) countries and Oman

A recent market survey shows that GCC countries current IT spending are $5 billion annually and are
expected to be doubled by 2010. Oman is emerging as one of the regions strong and fast growing markets.
Omans IT market is expected to grow from $230 million in 2005 to over $400 million by 2010 (GulfBase
2008) and (Inno Vest Group 2008). Currently, the average ICT spending in the region are between 2% to 4%
of GDP as compared to an average of 8% in developed countries. Also, PC penetration is below the world
average. The highest growth rate of internet usage access over the past six years is an encouraging sign in
the Middle East (Patel 2007). This shows that GCC countries including Oman have made strides in
advancing the development of ICT sector yet they have to go a long way ahead to reach to developed world
averages. Most of the ICT activities in these countries like in other developing countries are based on
governments initiatives and policies.
3.1 Use of ICT in Oman:

An Arab Gulf country in the Middle East, Oman has a population of 3.1 million. According to a recent (UN
2008) report, among some of ICT adoption indicators, Oman has 12.22 Internet per 100 users, 5.06 PC per
100 users, 69.6 Cellular subscribers per 100 users, 10.65 main telephone lines, and 0.58 broadband per 100
users. Oman has a Web Index of 0.4849 and ranked 52 in the world, an Infrastructure Index of 0.1559,
Human Capital Index 0.7659, and e-Government Index of 0.4691. Oman has an e-participation index of
0.2045 and rank 60 among the world. It is at the bottom of the list of the GCC countries appearing on most of
the above mentioned ICT indicator indices. Realizing the importance of ICT for the economic development of
the country, Omans government has placed a great emphasis upon creating a digital economy as key
drivers for the sustainable growth of the country. Oman is heavily investing in ICT as one of the building
blocks to diversify the economy. In September 2003 Oman established Knowledge Oasis Muscat (KOM
2008), a technology park as one of the initiatives taken to help develop a knowledge-driven economy, attract
investment, and to serve as an incubator for local start up companies in the ICT sector. Also, in 2006 the
government of Oman created an Information Technology Authority (ITA 2008) for developing a national IT
strategy, help facilitating and implementing an ICT infrastructure and overseeing the implementation of
Omans digital strategy. The government has encouraged private sector to open universities and technical
colleges in order to increase ICT literacy in the country. Government of Oman has established a Sanad
program to encourage entrepreneurship and develop SMEs by providing them necessary finance, guidance
and training. From the private sector a number of organizations are contributing in the growth of SMEs. For
example Shell has established a $10 million Intilaaqah Enterprise Fund to provide capital and on-going
support for SMEs in Oman. Microsoft has signed a Memorandum of understanding with Ministry of Education
to provide training in schools. Also, there are a number of other initiatives in place in order to transform
Oman into a digital society. It is hoped that all these steps will increase diffusion of ICT in Oman in general
including SMEs in near future.
According to an estimate of HSBC Middle East Bank there are only 15,000 to 20,000 SMEs in Oman
generating only 10-20% employment (Oman Economic Review 2007). These estimates show that there is a
significant potential for the SMEs in Oman to grow in terms of contribution to GDP and employment, and to
be competitive at regional and international levels.
Since there is no data available or a study carried out on the impact of ICT on SMEs in Oman, we were
interested to know about the present technological infrastructure, the reasons behind ICT investment, the
restrictions and barriers for adoption of ICT, and the implementation methods and benefits of ICT
investments. These aspects essentially formed the basis of our research.
www.ejise.com
127
ISSN 1566-6379
4. Objectives of the study

Realizing the importance of SMEs for the economy of Oman and the impact of ICT on improving the
performance of SMEs, data was collected on various aspects of ICT usage and utilization. The investigation
took place during January and April 2008 and is the first step towards an exploratory study on learning more
about the use and impact of ICT on SMEs in Oman. Through this study we aim to find answers to the
following questions:
1. What is the level of usage of ICT in terms of ICT infrastructure, internet connection, IT staff and types of
website used in SMEs?
2. What Enterprise software SMEs used?
3. What are the drivers for ICT Investment?
4. What percentage of budget is allocated to ICT?
5. What are long term business goals for investment in ICT?
6. How is competition among SMEs?
7. What are the barriers towards to ICT Investment?
8. What approach SMEs use to differentiate their business from competitors?
9. What are realities and perceptions about business benefits of ICT implementation?
10. What are out-sourcing trends?
11. What internal capabilities and processes SMEs have in place for managing ICT?
5. Research design
We performed the following tasks to conduct the research and determine the usages and effects of
information and communication technologies in SMEs in Oman:
1. A questionnaire was prepared based on the review of current literature and the Net Impact Canada 2006
study (Illuminas 2006) to determine the usages, effects and perceptions of Omani SMEs towards ICT.
2. The questionnaire contained 22 questions related to business aspects of the organization, ICT
infrastructure, use of internet, website, drivers for ICT investment, barriers to ICT adoption, competition,
and benefits of ICT.
3. The questionnaire was distributed to a number of SMEs in Muscat area, the capital of Oman. Fifty one
completed surveys were received from the companies who have adopted ICT. Those companies who do
not use any form of ICT are not included in our study. The questionnaires were completed by the
founder, general manager or accounts manager because of their ability and understanding of the issues
investigated in the questionnaire. In very few cases more than one individual representing the enterprise
filled out the questionnaire - this actually ensured preciseness.
4. Based on the 51 completed survey questionnaires, simple statistics were carried and logical inferences
were made to determine the general usages and effects of ICT on SMEs in Oman.
6. Research findings
Based on our definition of SMEs, 41% of the respondents can be classified as Micro Enterprises (less than
10 employees), 41% as Small Enterprises (between ten and fifty) and 18% Medium sized Enterprises
(between fifty and two hundred fifty). This shows that more than 80% of SMEs in our sample belong to Micro
and Small Enterprises.
Table 1: Types of SMEs based on number of Employees
Types of SMEs
Micro Enterprises
Small Enterprises
Medium Sized Enterprises
Percentage
41
41
18
6.1 ICT usage

We investigated IT infrastructure, Internet connection type, IT staff, usage of enterprise software, and type of
website used as a measure for ICT usage.
www.ejise.com
128
6.1.1 ICT infrastructure

Figure 2 shows that desktop, laptop or handheld computers are used by 100% of the surveyed SMEs. The
reason is that our sample is based on only those SMEs who have adopted some form of ICT in their
business (e.g. computers). During our survey we noticed that a large number of Micro Businesses (we
estimate 80% or more) do not use any form of ICT. It is not surprising as in the UK, 30% of micro businesses
do not use computers at all (Pritchard 2006). The reason seems to be the older generation of
Owner/Manager who are not used to ICT and/or are low educated foreign labor force (with no or little
knowledge of ICT). Business productivity software such as Microsoft Word, Excel and PowerPoint were used
by 82% of the surveyed firms. About 67% of the firms use enterprise software such as CRM, Inventory
Management, E-Commerce or ERP. Wired computer network solutions such as servers, routers and firewall
were utilized in 45% of the SMEs. Data storage and security solutions (such as file servers, storage area
network or web-based storage) were used by 25% of the surveyed firms, and wireless networking
technologies (such as access points and wireless routers) and network security solutions by only 14% of the
organizations surveyed.
Netw ork Security Solutions
14%
Data Storage Solutions
Business productivity
sof tw are
Wireless netw orking
hardw are/sof tw are
Solutions
25%
Enterprise Sof tw are
Applications
67%
82%
14%
45%
Wired netw orking

hardw are/sof tw are
100%
Desktop computers, laptops

or handheld devices
Usages
Figure 1: ICT infrastructure
6.1.2 Internet connection type

The type of internet connection in organizations largely indicates the required bandwidth and frequency or
purpose of usage. In Oman the SMEs have shown no reluctance in subscribing high-speed broadband
internet connection. This might be because of the relatively lower costs and higher speeds as compared to
the dial-up, but nevertheless, business heavily used internet as indicated by the surveyed enterprises.
Overall, 86% of SMEs in our survey used Internet. Figure 2 shows that 49% of the respondents use high
speed broadband (ISDN, ADSL, DSL), 35% use dial up connection, 2% use Satellite and 16% have no
internet connection. This is consistent with other studies. In a recent survey of SMEs in the UK, 78% of
SMEs use Internet in their business (Harindranath et al 2008).
16%
No Internet
Satellite Connection
Type
Very High Speed (T1, T2)
2%
0%
High-Speed Broadband (ISDN,

ADSL or DSL)
49%
Dial Up (less thank 56 Kbps)
35%
Percent
Figure 2: Type of Internet connection
6.1.3 IT staff
One may assume that small and medium sized enterprises often lack the supporting workforce needed, such
as IT staff, for the quality and quantity of the required support; the expertise of such staff are too little and
www.ejise.com
129
ISSN 1566-6379
basic. However some do have a special IT department with multiple supporting staff. The decisive factors
that determine the existence of such staff in SMEs around the world are the size of the firm and more
importantly the relative complexity of the adapted ICT solutions. The same is the case with SMEs in Oman
as around 65% of the surveyed firms do not have full-time IT staff (similar to 63% SMEs in Canada), and
35% of the firms have IT/IS departments with full-time IT staff.
Usage of enterprise software
Investment in ICT infrastructure enables businesses to take advantage of the large number of different
technologies available in the market. No matter how small the business and how basic the infrastructure,
businesses always look to automate and computerize the essential business functions to save costs and
time and to eliminate the need for support staff. Almost 84% of the surveyed firms use finance and
accounting enterprise software such as Tally and Peachtree. It was found that even some of the smallest
family run businesses used such software. Inventory management software is being used by 69% of the
firms and 31% of the enterprises use customer relationship management (CRM) software as illustrated by
Figure 3.
Resource Planning
14%
Supply Chain Management

Human Resource
Management
Customer Relations
Management
Type of System
e-Commerce
4%
8%
16%
31%
69%
Inventory Management
84%
Finance/Accounting
Usage (%)
Figure 3: Enterprise system usage

It was observed that organizations that invested in enterprise software are of medium size, in terms of the
size of workforce or annual turnover within the SME category, or they are micro enterprises with high
inventory turnover. For example, one small retail business run by a single person (owner) used enterprise
software for automating accounting, inventory and customer related business processes. Figure 3 shows
that a majority of SMEs in Oman are utilizing basic Enterprise software such as finance/accounting and
inventory management in their businesses.
6.1.4 Types of website used

Unlike SMEs in developed countries, the SMEs in Oman have not managed to utilize and use commercial
websites. Results shows that only 2% of the firms have commercial websites which helps them reach new
customers. As for informational websites, 27% of the surveyed enterprises run websites which simply
introduce the business and publish their contact information. About 73% of the businesses had no website,
the main reason was lack of internal staff and high maintenance costs in the long term. This shows that
SMEs are not fully utilizing Internet.
This shows that ICT usage within SMEs who have adopted ICT is relatively moderate in common
technologies (such as desktop, laptop, productivity software etc) but limited in the more sophisticated
technologies such as wireless, data storage and network security solutions (in those companies who have
adopted ICT). About 86% of surveyed SMEs use Internet but only 27% have a website and only 2% have
commercial website. Also, 65% do not have full time ICT staff. The results seems to be very similar to other
studies (Illuminas 2006) and (WestFocus 2007)
6.2 Investments in ICT

Many of the decision makers in small and medium firms are not always aware of relevant ICT that could
revolutionize their business. This in itself is a barrier to ICT investment. As for those enterprises that have
already invested and are willing to continue, what really encourages them to do so? And what is the
proportion of their investment? Also, what is the nature and true reason behind their investments? We have
used amount of budget dedicated to ICT, drivers for ICT Investment, number of competitors, long term ICT
www.ejise.com
130
investment goals and organizations strategy for competitive advantage as key elements to learn about
reasons for ICT investment.
6.2.1 IT budget
Figure 4 shows that 70% of the SMEs assign less than 10% of their annual budget to ICT investment, only
2% of the firms invest between 10% and 20%, and 8% of the surveyed firms assign greater than 40% of their
budgets to ICT.
8%
2%
20%
<10%
Between 10% and 20%
Between 20% and 40%
>40%
70%
Figure 4: Proportion of budget assigned to ICT
6.2.2 ICT investment drivers

Regardless of the relative proportion of budget invested in ICT certain stimuli exist that encourage and push
decision makers in SMEs to invest in ICT. The main driving forces for ICT Investment are to provide better
and faster customer service (65%), and to stay ahead of competition (69%). These findings are in consistent
with WestFocus that found three biggest perceived benefits of ICT usage were keeping up with the
competition, faster response to customers and improved quality. Figure 5 shows stimuli for ICT investments
in SMEs.
69%
Demands of stakeholders
65%
Better and f aster customer support
and services
25%
To stay ahead of competition
Percent
45%
Follow ing the strategy set by top

management
Advice f rom consultants
Demand of your suppliers
6%
6%
Government requirements
2%
Reasons
Figure 5: ICT investment drivers
6.2.3 Number of competitors

Most of the business decisions are based on the competition in the market. Figure 6 shows that 50% of the
enterprises have more than twenty direct competitors. About 16% of SMEs have between ten and nineteen
competitors, 16% between five to nine competitors and 14% have between one to four competitors. This
shows that there is reasonably strong competition among SMEs in Oman, and perhaps that is one of the
main reason that SMEs would like to be more competitive by adopting ICT.
www.ejise.com
131
ISSN 1566-6379
4%
14%
16%
1-4
5-9
10-19
20+
50%
Don't Know
16%
Figure 6: Number of direct competitors
6.2.4 Long term goals for ICT investment

Most of the surveyed firms long-term plan with regards to their ICT investment was to increase their market
share, grow their sales revenue and cut costs and expenses. About 86% of the firms expect to increase
sales revenue, 67% expect to increase their market share and about 55% expect to cut costs as shown in
figure 7.
86%
67%
Percent
55%
Growth of sales revenue

Increase market share
Cutting costs
Strategy
Figure 7: Long term plans

The above results show that Omani SMEs have strong competition and are devoting a reasonable
percentage of their budget to ICT in order to provide better and faster customer service. Also to realize
benefits of ICT investment, SMEs expect to increase sales revenue and market share as well to reduce
costs. Most of the SMEs aim to provide highest quality products and services to their customers as well as
establishing long term relationships with customers. (Harindranath et al 2008) also found increasing sales
and reducing costs as main reasons for investing in ICT.
6.3 Competitive strategy

A common way of achieving business goals is to differentiate ones business from the competition. The
surveyed SMEs have chosen the approach of providing the highest quality products and services (80%) to
their customers as the principal method for differentiating as well as establishing long-term relationships with
customers (47%), as illustrated in Figure 8.
www.ejise.com
132
Percent
27%
47%
80%
31%
By providing the most

innovative products or
services
By establishing the best
relationship with
customers
By providing the highest
quality products or
services
By providing the lowest
cost products or services
Approach
Figure 8: Approaches to achieve competitive advantage
6.4 Barriers to ICT investment

While some of the SMEs in Oman are clearly aware of ICT and its benefits, there exist certain restrictions
and barriers to ICT investment. Figure 9 shows that 75% of the firms feel that a lack of necessary internal
skills is a major barrier. This is consistent with other studies that SMEs do not have enough human
resources (Wymer and Regan 2005). There has been a recent increase in technological colleges and the
general investment by the Omani government in the ICT industry to overcome shortage of IT staff. About
63% feel that the monetary costs of ICT solutions and implementation are too high. (Harindranath et al 2008)
also identified cost as the single most factor threatening future investment in ICT. Almost 63% of the
decision-makers within the surveyed firms feel that there is not enough information available at their disposal
about relevant and effective technologies. This finding also confirms findings of (Harindranath et al 2008),
(Chibelushi 2008) and (WestFoucs 2007) study who found concerns over costs and uncertainty over the
business benefits-followed by a lack of internal expertise. This shows that there is a need for free advice and
relevant information for SMEs. Of the respondents, 31% feel they simply have no time to implement the
projects. About 47% of the firms are uncertain about retaining their ICT investment and 18% of the managers
feel there is not enough support from the top-management in the firms. Other barriers identified include:
government regulations and requirements (6%), and bad experiences in the past (8%). A few of the firms
also complained about lack of infrastructure in certain areas in the country. One business specifically
complained about the unavailability of internet access in some of the areas far away from Muscat, the capital
of Oman. Figure 9 shows some of the main barriers to ICT adoption of SMEs in Oman.
75%
63%
63%
Monetary costs of implementation

Time to implement ICT project
Percentage
47%
Lack of necessary internal skills
31%
Uncertain about retain on investment
18%
Lack of available information about relevant

technologies
Lack of top-management
support/direction/planning
Barrier to Investm ent
Figure 9: Barriers to ICT investment
www.ejise.com
133
ISSN 1566-6379
6.5 Realization of business performance improvements

We solicited opinions about the actual and expected benefits of ICT investment in improving the business in
a series of questions.
Table 2 shows that only 27% SMEs have realized business impact beyond their geographical area. The
reason is that only 27% of SMEs have website. It is worth noting that 80% have realized reducing costs due
to ICT, while 53% have improved revenue and 57% have an impact on customer relationship due to ICT.
Almost all of the SMEs are more positive of future expected benefits of ICT implementation and investment.
This shows that those SMEs who have adopted ICT have realized benefits due to ICT and are very positive
in continuing to invest and harvest those benefits. (Harindranath et al 2008) also found that of the SMEs
were satisfied with their ICT investment and perceived their ICT investment as offering good value for
money.
Table 2: Actual and expected business performance improvements due to ICT
Actual
Business Performance
Yes
No
27%
73%
80%
20%
53%
47%
57%
43%
Impact of ICT
Beyond their geographical area
In cutting costs
On revenue
On customer relationship
Expected Business
Performance
Yes
No
41%
59%
84%
16%
55%
45%
67%
33%
6.6 ICT implementation and outsourcing

One of the solutions to solve the problem of unavailability of internal skills many organizations have the
option of outsourcing their ICT related work. Therefore, what we know as outsourcing has become one of
the most common practices performed by almost all organizations, both private and public, small and large,
in order to remain competent within their fields of business or work.
6.6.1 Proportion of activities outsourced

Majority of the SMEs that participated in this research outsourced a portion of their ICT activities and work.
Figure 21 shows that 57% of SMEs outsourced more than 50% of their activities, 14% outsourced between
10 and 25 percent of their activities, and 14% outsourced less than 10% of their ICT activities. This relates to
the shortage of IT staff, lack of internal skills, lack of relevant information and guidance on relevant
technology solutions and high costs of ICT mentioned earlier sections. (Harindranath et al 2008) study who
found that 50% of the firms in their survey used external consultants in ICT matters.
8%
None
Activity
57%
>50%
Between 25% and 50%
8%
Between 10% and 25%

<10%
14%
14%
Proportion Outsourced
Figure 10: Proportion of ICT activities outsourced
www.ejise.com
134
6.6.2 Resources used for ICT implementation

It is important to know what ICTs were adopted and used within SMEs, but it is also important to know which
internal or external resources were used to implement these technologies. Most of the enterprises surveyed
used internal resources to implement basic technologies such as the installation and setup of desktop or
laptop computers, data storage hardware or business productivity software. As the technologies become
more sophisticated, like the use of enterprise software applications, wireless networking or the use of mobile
phone applications, the firms tended to use external resources. Figure 22 shows that 33% organizations use
internal resources for Desktop or laptops, as well as 24% for data storage solutions and 24% for business
productivity software. About 59% of the SMEs outsourced more sophisticated technologies such as
enterprise software applications. Also, it seems that some of the organizations lack internal capacity for basic
technologies such as Desktop or laptops (47%), business productivity software (41%) and data storage
(35%), as they used outsourcing for these activities
A number of firms use both internal and external resources to implement some of the technologies. It was
observed that SMEs often use external resources to initially implement the technology, and then tend to use
internal resources to maintain and upgrade the implemented technologies.
47%
41%
Internal
35%
33%
External
Both
24% 24%
8%
12%
8%
4%
10%
10%
Desktop PCs,
laptops or
handheld PCs
Business
productivity
software
Wireless
networking
solutions
Data storage
solutions
2%
18%
16%
10%
10%
6%
Enterprise
software
applications
20%
Networking
hardware/software
solutions
24%
Network security
services
Trend Percentage
59%
ICT Solutions
Figure 11: Resourcing trends Internal vs. external

This shows that most of the SMEs lack skills and resources in Computers, ERP, Business software, data
storage and network solutions. They look for external help in almost all areas of ICT. This emphasizes need
for ICT training for SMEs. (Harindranath et al 2008) and (Chibelushi 2008) also found that lack of ICT
expertise as one of the main barrier in their study.
7. Conclusions
We found that ICT usage within SMEs in Oman is moderate in common technologies, but limited in the more
sophisticated technologies such as wireless, data storage and network security solutions. It was noticed that
Omani SMEs are taking a comprehensive approach to their ICT investment focusing on both strategic and
operational aspects of their business. The results of our study show that Omani SMEs are making
reasonable investment in ICT and that there is a modest competition among the SMEs. The main driving
forces for ICT investment was to provide better and faster customer service, to stay ahead of competition
and following top management strategy. The competitive strategy for the majority of SMEs was to provide
high quality products and services to their customers and to establish long term relationships with customers.
www.ejise.com
135
ISSN 1566-6379
Nearly half of the respondents have realized business benefits of ICT adoption such as better customer
relationships, increase in revenue and in reducing costs. Two thirds of the respondents are very positive
about increasing their business performance in the future.
With regard to barriers to ICT investment, three fourths of the firms feel that a lack of necessary internal skills
is a major barrier. More than half of the respondents feel that the costs of implementation are too high. Lack
of availability of relevant information and advice on suitable and effective technologies is also one of the
major barriers. Other barriers include SMEs having no time to implement ICT projects, lack of top
management support, bad experience in the past and government regulations and requirements. These
findings are consistent with other studies e.g. (Harindranath et al 2008). This emphasizes the need for more
training facilities in ICT for SMEs, measures to provide ICT products and services at an affordable cost, and
availability of free professional advice and consulting at reasonable cost to SMEs. Our findings therefore
have important implication for policy aimed at ICT adoption and use by SMEs.
More than half of the participants outsourced over 50% of their activities. This can be related to lack of inhouse capabilities in ICT identified as a major barrier. These results also confirm findings of (Harindranath et
al 2008) and (Chibelushi 2008) and re-emphasize the need of ICT training for SMEs.
Overall, it seems that only a small number of SMEs in Oman are aware of the benefits of ICT adoption. The
findings of our research show that the SMEs lack necessary ICT knowledge and skills and mechanism to find
and receive advice and support. There is a need for more focus and concerted efforts on increasing
awareness among SMEs on the benefits of ICT adoption in order for SMEs to be more productive and
competitive. Also, there is a need for providing affordable ICT products, services, solutions and relevant
professional advice for SMEs. There is a need for government and professional trade organizations (such as
Chamber of Commerce and Industry) to address the gaps and issues identified in this study. The findings of
this research will provide a foundation for future research and will help policy makers in understanding the
current state of affairs of the usage and impact of ICT on SMEs in Oman.
8. Limitations of the research and directions for future research

This study was a preliminary exploratory study to learn about a few selected aspects of ICT adoption in a
GCC country, Oman. There are number of issues such as legal, regulatory, interventions from the
government, just to name a few, in the adoption of ICT that needs further investigation. A detailed study
should take a more comprehensive approach considering a wide range of areas of ICT adoption. These
results are based on a small sample of 51 SMEs, out of that 80% belongs to Micro and Small enterprises.
Data was collected from SMEs who use some form of ICT in their business. Those organizations that do not
use computers were excluded from the study. Our conservative estimate is that currently, more than 80% of
Micro businesses do not use any form of ICT for reasons mentioned earlier in the paper. The results show a
general trend and practices of the use and impact of ICT on SMEs in Muscat, the Capital of Oman. The
results should be interpreted or used with these perspectives in consideration. A larger sample is needed to
further validate these trends.
References
Baldwin, J.R., Jarmin, R.S., and Tang, J. (2001) The Trend to Smaller Producers inManufacturing in Canada and the
US, Statistics Canada Working paper.
Caldeira, M.M., and Ward, J.M. (2002) Understanding the Successful Adoption and Use of IS/IT in SMEs: an
Explanation from Portuguese Manufacturing Industries, Information Systems Journal, Vol. 12, No. 2, pp 121-152.
Chibelushi, C. (2008) ICT industry challenges in adopting ICT: a case study from the West Midlands, UK, in
proceedings of the 2008 International Conference on Information Resource Management, May 18-20, Niagara Falls,
Canada.
Clarke, J. (2004) "Information Systems Strategy and Knowledge-based SMEs in the Australian Biotechnology Industry:
Does IS need to reorient its Thinking?" 15th Australian Conference on Information Systems, Hobart, Australia, 1-3
December.
Dawn, J., Podonik, P., and Dhaliwal, J. (2002) Supporting the E-business Readiness of Small and Medium-Sized
Enterprises: Approaches and Metrics, Internet Research Vol. 12, No. 2, pp139-195.
Doczi, M. (2002) Information and Communication Technologies and Social and Economic Inclusion", Information
technology Policy Group, Ministry of Economic Development, Manatu, Ohanga, NZ, March.
Duan, Y., Mullins, R., Hamblin, D., Stanek, S., Sroka, H., Mavhado, V., Araujo, J. (2002) Addressing ICTs Skill
Challenges in SMEs: Insights from three country investigations, Journal of European Industrial Training, Vol. 26, No.
9, pp. 430-441.
www.ejise.com
136
Dutta, S., and Coury, M, E. (2003) ICT Challenges for the Arab World, Chapter 8, in Dutta, S., Lanvin, B. & Paua, F.
(ed.) The Global Information Technology Report 2002-2003: Readiness for the Networked World, World Economic
Forum/Oxford University Press, New York, pp116-131.
Fink, D. and Disterer, G. (2006) International Case Studies: To what extent is ICT infused into the Operations of
SMEs?, Journal of Enterprise Information Management, Vol. 19, No. 6, pp 608-624.
Gregor, S., Fernandez, W, Holtham, D, Martin, S, Vitale, M, & Pratt, G. (2004) "Achieving Value from ICT: Key
Management Strategies", Department of Communications, Information Technology and the Arts, ICT Research
Study, Canberra.
GulfBase (2008) GCC Common market set to enrich ICT sector [online]
http://www.gulfbase.com/site/interface/NewsArchiveDetails.aspx?n=49939 [11 March 2008]
Harindranath, G., Dyerson, R. and Barnes, D. (2008) ICT Adoption and Use in UK SMEs: a Failure of Initiatives? The
Electronic Journal Information Systems Evaluation Vol. 11, No. 2, pp. 91-96.
Houghton, K.A. and Winklhofer, H. (2004) The Effect of Website and E-commerce Adoption on the Relationship
between SMEs and Their Export Intermediaries, International Small Business Journal, Vol. 22, No. 4, pp. 369-388.
Illuminas (2006) Net Impact Canada 2006, [online], http://www.netimpactstudy.com/ca/ {20 March 2008]
Inno Vest Group (2008) The Technology Small and Medium-Sized Enterprises (SMEs) in Middle East Region, [online],
http://www.theinnovestgroup.com/43.html [12 March 2008]
ITA (2008), Information Technology Authority, Oman [online] http://www.ittf.gov.om/english/about_ita.html [15 March
2008]
Kapurubandara, M., and Lawson, R. (2006), Barriers Adopting ICT and E-commerce with SMEs in Developing
Countries: An Exploratory Study in Sri Lanka, CollECTeR 06, 9 December, 2006, Adelaide, [online],
http://www.collecter.org/archives/2006_December/07.pdf [13 March 2008]
Koellinger, P. (2006) Impact of ICT on Corporate Performance, Productivity and Employment Dynamics, Special Report
No 01/2006, European Commission Enterprise & Industry Directorate General, Berlin.
KOM 2008, Knowledge Oasis Muscat [online] http://www.kom.om/ev.htm [12 March 2008]
Lal, K. (2007) Globalization and Adoption of ICTs in Nigerian SMEs, Science, Technology Society, Vol.12, No. 2,
pp217-244.
Lange, T., Ottens, M., and Taylor, A. (2000) SMEs and Barriers to skills development: A Scottish Perspective, Journal
of European Industrial Training, Vol. 24, No. 1, pp. 5-11.
Lawson, R., Alcock, C., Cooper, J., Burgess, L. (2003), Factors Affecting the Adoption of Electronic Commerce
Technologies by SMEs: An Australian Study, Journal of Small Business Enterprise and Development Vol. 10, No. 3,
pp. 265-276.
Love, E.D., Irani, Z., Edwards, D.J. (2004), Industry-centric Benchmarking of Information Technology Benefits, Costs
and Risks for Small-to-Medium Sized Enterprises in Construction", Automation in Construction, Vol. 13, No. 4, pp.
507-524.
Lucchetti, R. and Sterlacchini, A. (2004) The Adoption of UCT among SMEs: Evidence from an Italian Survey", Small
Business Economics Vol. 23, No. 2, pp.151-168.
Morikawa, M. (2004) Information Technology and the Performance of Japanese SMEs, Small Business Economics Vol.
23, No. 3, pp.171-177.
Mutula, S.M., and Brakel, P. (2006) "E-readiness of SMEs in the ICT Sectors in Botswana with Respect to Information
Access", The Electronic Library, Vol. 24, No. 3, pp.402-417.
Oman Economic Review, 2007, SME Forum Making Life Easy, [online].
http://www.oeronline.com/php/2007_dec/sme_forum.php [8 March 2008]
Parker, C.M., and Castelman, T. (2007) "New Directions for Research on SME-eBusiness: Insights from an Analysis of
Journal Articles from 2003 to 2006", Journal of Information Systems and Small Business, Vol. 1, No. 1/2, pp. 21-40,.
Patel, F. (2007) The Arab business worlds ICT opportunity, Arabian Business World (23 August 2007).
Pritchard, S. (2006) How can so many businesses cope without computers Financial Time London (19 Sep. 2006).
Schubert, P., and Leimstoll, U. (2006) The Importance of ICT: An Empirical Study in Swiss SMEs in 19th Bled
Conference eValues, Bled, Slovenia, June 5-7.
Schubert, P., and Leimstoll, U. (2007)a Importance and Use of Information Technology in Small and Medium-Sized
Companies, Electronic Markets, Vol. 17, No. 1, pp. 38-55.
Schubert, P., Fisher, J., and Leimstoll, U. (2007)b ICT and Innovation in Small Companies 15th European Conference
on Information Systems, St. Gallen, Switzerland, 1226-1239, 7-9 June.
Shiels, H., McIvor, R., and OReilly, D. (2003) Understanding the Implications of ICT adoption: Insights from SMEs,
Logistics Information Management, Vol.16, No. 5, pp. 312-326.
Smallbone, D., North, D., Vickers, I., and Roper, S. (2001) Innovation and the Use of Technology in Manufacturing
th
Plants and SMEs: An Inter-Regional Comparison, Proceedings 24 ISBA National Small Firms Conference:
Exploring the Frontiers of Small Business (Hinckley, Leicestershire: ISBA), pp. 289-308.
Ssewanyana, J., Busler, M. (2007) Adoption and Usage of ICT in developing Countries: Case of Ugandan Firms,
International Journal of Education and Development using ICT, Vol. 3, No. 3, [Online] journal, available at
http://ijedict.dec.uwi.edu/viewarticle.php?id=349 [March 13 2008]
Stroeken, J.H.M. (2001) The Adoption of IT by SMES: The Dutch Case, Journal of Enterprising Culture, Vol. 9, No. 1,
pp. 129-152.
Temtime, Z.T., Chinyoka, S.V., Shunda, J.P.W. (2003) "Toward Strategic Use of IT in SMEs: A Developing Country
Perspective", Information Management & Computer Security Vol. 11, No. 5, pp. 230-237.
UN (2008) E-Government Survey 2008, From e-Government to Connected Governance, Doc #
ST/ESA/PAD/SER.E/112, United Nations, New York
www.ejise.com
137
ISSN 1566-6379
UNCTAD (2004) ICT as an Enabler for Growth, Development and Competitiveness, Plenary session, June 17, [online],
http://www.unctadxi.org/templates/Event____69.aspx [13 March 2008]
WestFocus (2007) Abandon Heroes ICT Adoption and Use in SMEs [online]
http://www.westfocus.org.uk/SiteManager/UserFiles/downloads/14_Abandoned_Heroes_Project_Report_final.doc [5
June 2008]
Wymer, S.A. and Regan, E.A. (2005) Factors influencing e-commerce adoption and use by small and medium
businesses Electronic Markets, Vol. 15, No.4, pp. 438-453.
Yeh, Quey-Chen, and Chang, A, J-T (2007) "Proliferation of Computers among Taiwanese SMEs", International Journal
of Entrepreneurship and Innovation Management, Vol. 7, No. 1, pp 51-73.
www.ejise.com
138
The Project Objectives Measurement Model (POMM): an

Alternative View to Information Systems Project Measurement
Corlane Barclay1 and Kweku-Muata Osei-Bryson2
University of the West Indies, Jamaica
2
Virginia Commonwealth University, USA
CLBarclay@gmail.com
KMOsei@vcu.edu
1
Abstract: The information systems (IS) project management profession has been faced with numerous socio-technical
challenges. As part of its analysis, research has expressed discontent with the traditional measures used to assess the
success or failure of these projects, i.e. conformance to time, schedule and specification requirements espoused by the
project management (PM) standard bodies. Despite this, research has also revealed that industry continues to place high
reliance on this approach in determining the outcome of their projects. These developments imply, in part, a
misalignment between research and practice and a scarcity of appropriate measurement tools that are aligned to the
realities of different project contexts. The research presents a Project Objectives Measurement Model (POMM) that
attempts to address some of these concerns through the development of project measures that are aligned to key project
stakeholders values and objectives within the unique project contexts. It is argued that objectives are the key
performance criteria of the project hence measures must be aligned to these criteria and formal procedures should be in
place to assure that these objectives and measures are carefully developed and reflective of the persons to which the
project matters, the stakeholders. The POMM is grounded on several principles of the Value Focused Thinking (VFT)
and Goal Question Metric (GQM) techniques. The evaluation of the proposed model was performed in two parts: a team
of industry experts examined the principles of model and provided feedback on its practicability to practice, and a case
study of a Caribbean educational institutions IS graduate programme development was used to illustrate the procedures
of the model.
The research provides theoretical and practical implications for IS evaluation particularly within the project management
and performance measurement domains. The research aims to extend the debate on suitable evaluation methods for IS
projects while providing project practitioners with an alternative approach that can enhance their decision making
processes during the life of the project.
Keywords: IS project; project objectives measurement model (POMM); success criteria; IS project management;
Caribbean
1. Introduction
Information systems (IS) projects play an important, often strategic role in contemporary organisations. Given
this prominence, executives and other project stakeholders have a vested interest in assessing the real
value that these projects provide (Brynjolfsson, 1993, Melville et al., 2004). It is therefore important to find
appropriate methods to evaluate them, including the value that it provides to relevant stakeholders. Current
analysis indicates several challenges with regards to current practices. There is growing disenchantment
with the traditional methodology (i.e. conformance to time, cost and scope requirements) which has been
described as an incomplete measurement paradigm (Atkinson, 1999, Atkinson et al., 2006, Wateridge,
1998). Despite its perceived limitation this approach continues to be pervasive in practice (White and
Fortune, 2002, KPMG, 2005). A compounding concern is the varied perception of performance, as success
may mean different things to different stakeholder which may lead to disagreement on whether a project is
successful or not (Shenhar et al., 2001, Shenhar et al., 2002).
Against this background, we argue that an effective evaluation approach for IS projects should consider
active alignment of stakeholders value to the measures. The research explores an alternative measurement
system for assessing IS project performance that extends the traditional approach. We therefore propose the
Project Objectives Measurement Model (POMM) which adapts the principles of the Value Focused Thinking
(VFT) method (Keeney, 1992) to elicit and structure stakeholders project values and objectives, and the
Goal Question Metric (GQM) method (Basili and Weiss, 1984) to derive measures that are linked to the
identified project values and objectives. The POMM can be used to provide guidance to practitioners in the
design of project metrics/measures that are closely aligned to the realities of their IS projects in a manner
that transcends the restrictive frames of traditional approaches. Further it aims to enable project
stakeholders to be better equipped to see the project is, identify missing links or inconsistencies in project
design, and take appropriate corrective actions where necessary, thus increasing the likelihood of
achievement of the objectives or project performance criteria.
ISSN 1566-6379
139
Barclay, C. and Osei-Bryson, K-M The Project Objectives Measurement Model (POMM): an Alternative View to
Information Systems Project Measurement. The Electronic Journal Information Systems Evaluation Volume 11 Issue 3
2008, pp. 139 - 154, available online at www.ejise.com
With continued calls to extend the conceptual foundations of project management (PM) through the analysis
of similar events in other disciplines and increase the alignment with practice (Engwall, 2003, Winter et al.,
2006b, Smithson and Hirschheim, 1998) the study seeks to extend the knowledge base in the discipline. The
research uses empirical investigation to help demonstrate our claims. First, a preliminary evaluation of
POMM is performed by a team of IS PM experts and stakeholders. Second, a case study illustration of
POMM using an IS graduate programme project is presented. Findings indicate that POMM has credence in
practice and can be suitable in assisting practitioners in strengthening their evaluation process.
2. Research Background
2.1 Trends in evaluating projects
An appreciation of the full context within which the project is being performed will assist those involved in
project management to deliver a project (APM, 2006), as these activities continue to play key role in
organisational change. Projects are unique, transient endeavors undertaken to achieve a desired outcome
(APM, 2006) including a unique product, service or results (PMI, 2004) and refer to single or multiple
endeavours. Assessing outcome of these projects is of extreme importance to the key stakeholders involved
(Cleland, 1986). However, with the continued problems with assessing the value of IS (Irani, 2002), the true
contribution of IS projects are often not realized or identified. Thus, it is essential to find more insightful
analysis of project performance or outcome to assess where we are and whether we need corrective action
to get back on track.
A review of key project evaluation literature revealed that the primary responses to examining the
performance of projects have been to develop alternative success criteria to assess these projects (Atkinson,
1999, Morris and Hough, 1987, Nelson, 2005), and critical success factors (Fortune and White, 2006,
Belassi and Tukel, 1996, Pinto and Slevin, 1987, Shenhar et al., 2001, Shenhar et al., 2002) while others
have focused on the business value contribution of these investments (Kaplan and Norton, 1992, Kumar,
2003, Fitzgerald, 1998). In an analysis of the literature over the last forty (40) years, Jugdev & Muller (2005)
showed the evolution of our understanding within the framework of the project and product life cycles in the
determination of outcome perspectives. The four (4) evolving research themes were categorized into project
implementation and handover, critical success factor (CSF) lists, CSF frameworks and strategic project
management paradigms (Jugdev and Muller, 2005).
Table 1: Summary of success factors & criteria
Key
Categorization
Project
Management &
Project Team
Management &
Executive
Client and Other
Stakeholders
Project product
or service
Preparation for
the Future
Outcome Perspectives
Literature
Minimization of project cost and project duration; strong

project commitment; communication; monitoring and
feedback; personnel & competence; planning,; clear
project objectives; conformance to budget, time, scope
requirements; project functionality, project efficiency,
Top management involvement, politics
(Atkinson 1999); (Bryde et al. 2005);

(Freeman et al. 1992) (Morris et al. 1987);
(Pinto et al. 1987); (Nelson 2005), (Shenhar
et al. 2001; Shenhar et al. 1997; Shenhar et
al. 2002)
(Pinto et al. 1987); (Morris et al. 1987)
Satisfaction; endorsement; acceptance; user

involvement; utility; use; safety; impact on customer;
customer service; increased responsiveness
Belassi et al. (1996); Bryde et al. (2005);

Kumar (2005) ; Pinto et al. (1987); (Standish
1994); Lim et al. (1999), Nelson (2005),
Morris et al. (1987) ;
Dvir et al. (1992); Lim et al. (1999) ;
Fitzgerald (1998) ; Freeman et al. (1992)
New product or market; safety; commercial

performance; technical performance business and
direct success; financial rewards; implementability;
flexibility
Value, personal growth, learning, readiness for the
future
Bryde (2005), Nelson (2005) ; Freeman et al.

(1992) ; Kaplan & Norton (1992)
2.2 Managing stakeholders expectations

Success may mean different things to different stakeholders (Shenhar et al., 2001, Shenhar and Levy, 1997,
Belassi and Tukel, 1996) which may lead to difficulties in managing these perspectives (Agarwal and
Rathod, 2006) and determining the success of the project outcome. This has interesting implications,
particularly the opportunity to synthesize the diverse views and transform these into suitable measurement
frameworks, hence, facilitating richer analysis rather than simple listing of objectives.
www.ejise.com
140
Corlane Barclay and Kweku-Muata Osei-Bryson
The stakeholders group is diverse, and may include individuals that are internal and external to the project
and organisation such as contractor, sponsor, project team and client (PMI, 2004). They are typically most
involved in the project and have a vested interest in its outcome or contribution (PMI, 2004). These
stakeholders may have certain expectations and consequently engage in behavior that may constructive or
destructive (Bourne and Walker, 2006). Importantly, Courtney (2001) noted that open, honest, effective
dialogue among stakeholders is a critical aspect in the development of multiple perspectives, which implies
that reaching effective measurement tool is made easier open honest and effective dialogue about
stakeholders viewpoint.
2.3 Rethinking the profession

An interdisciplinary colloquium in the UK met to discuss and analyse initiatives that may enrich and extend
the current project methodologies to better meet these and other contemporary challenges (Winter et al.,
2006b) as it recognized that bridging the process of academic and practitioner perspectives is still imperative
(Winter et al., 2006a). The need for new and better ways to think about projects and their management
(Crawford et al., 2006) (Cicmil et al. 2006) revealed that current views do not adequately explain the richness
of what actually happens in contemporary project environments This complements the view that current
practice does not sufficiently address sources of uncertainty within the project and thus more sophisticated
efforts are needed (Atkinson et al., 2006). The participants therefore called for increased focus on concepts
and theories that resonates with the realities of practice thereby providing practitioners with practical
concepts and improve alignment with contemporary thinking (Winter et al., 2006b) and recommended that
PM research embraces theory about practice, theory for practice and theory in practice (Winter et al.,
2006b).
3. Towards a new methodology for practice

An objective is defined as a lens or system of lenses that forms an image of an object (Merriam Webster
Dictionary, 2008). In this case, the stakeholders lens forms an image of the project that results in the
perception of its performance. Hence, project objectives therefore can be considered the stakeholders
criteria through which performance outcome perceptions are based. Assessing the outcome of the projects
based on these factors appears to better match the realities of practice. It is therefore critical that the
objectives are robustly defined and reflects the views of the key participants.
3.1 Value Focused Thinking (VFT) approach

Understanding what is important to key stakeholders can enhance the decision-making process in the
project. The VFT is a decision technique developed by Keeney (1992), it identifies the values of the
stakeholders in a given activity. The technique has been applied successfully in other research, and is a
proven methodology that has been used in various disciplines to aid in the analysis of strategic decisions
(Sheng et al., 2005). The VFT is useful in uncovering hidden strategic objectives of diverse managerial
processes, and has been used successfully in the operations management disciple to help chart a clear
decisive path for the fundamental objectives and values of the activities thereby providing an unobstructed
view to carefully assessing performance of organisational activities (Keeney, 1992, Keeney, 1996). Keeney
(1996) suggested that the VFT enriches the decision making process. Further, while it is important to identify
objectives, simply listing objectives is shallow as there is need for greater depth, clear structure, and a sound
conceptual base in developing objectives for strategic decision contexts (Keeney, 1996).
Figure 1: Steps in the VFT
www.ejise.com
141
ISSN 1566-6379
The application of the VFT technique includes the following steps (Keeney, 1992, Keeney, 1996):
Develop list of objectives. This involves identifying the values and converting values to objectives.
Values are those principles that encompass what a person care about or value in a specific situation. An
objective is characterized by three features: a decision context, an object and a direction of preference.
This essentially means explicating the objective within its context based on the nature of the problem
and determining exactly what the stakeholder is ultimately trying to achieve. It is also possible to derive
more than one objective from a specific value statement (Drevin et al., 2007).
Structure objectives. Objectives are classified into two types: (1) fundamental objectives - the end that
the decision-maker values in a specific decision context, and (2) mean objectives the methods to
achieve the ends which are context dependent. This implies that consideration of the particular nature or
purpose will determine how the fundamental and mean objectives are formed.
Develop means-ends network. To perform this step, asking why each objective is important will help to
distinguish between fundamental and means objectives which can be represented by the means-end
objective network. Aided with this tool, decision makers are better able to see the relationship among the
project objectives and can make more informed decisions in the project
3.2 GQM approach

Subsequent to the identification of core objectives of the project, the identification of suitable quantifiable
metrics aligned to the objectives is essential. The GQM approach is a metric generation technique that
develops performance metrics aligned with the goals (objectives) of the activities (Basili and Weiss, 1984). It
utilizes a top-down method for the identification of metrics needed for certain goals by asking questions
linked to these goals. It is a practical approach to bounding the measurement problem as it allows
organisations to focus on its own context and culture. The method asks, how do you decide what you need
to measure in order to achieve your goals (Basili and Weiss, 1984)? Applying these principles into the IS
project evaluation context strengthens the generation of stronger metrics and measures associated with the
project objectives.
The scope of the GQM application in this research is primarily to identify the metrics subsequent to the
application of the VFT to identify project objectives. These steps include (Basili and Weiss, 1984, Solingen
and Berghout, 1999):
1. Formalize measurement goals
2. Identify quantifiable questions
3. Define the measures to be used
4. Prepare plan for implementing and interpreting the measures
4. The project objectives measurement model (POMM)

The Project Objectives Measurement Model (POMM) involves the elicitation of objectives and measures that
reflect the strategic and tactical vision of the project from the perspectives of its multiple stakeholders. Three
key questions are reflected throughout the framework:
1. Do the project measures reflect the fundamental objectives identified?
2. Do the project objectives reflect the project contexts?
3. Does the evaluation process reflect the realities of the project?
www.ejise.com
142
Figure 2: The POMM conceptual framework

POMM involves the following several steps:
1. Identify key stakeholders of the project, taking into consideration the roles involved, the organisations or
personnel that may be impacted by the project and its results
2. Elicit project values and objectives for each key stakeholder, including standard objectives relating to
time, cost and scope
3. Apply VFT method to determine the fundamental and means project objectives
4. Prioritize project stakeholder fundamental objectives
5. Develop, review and refine (where necessary) the project means-end network
6. Apply GQM method to elicit project measures
7. Develop, review and refine (where necessary) the project objective-measure network
8. Implement, monitor and take corrective actions throughout the project
9. Determine the cumulative outcome of the project
The Objective-Measure Network (Figure 3) is an output of these steps. It is an extension of the Means-End
Network (Keeney, 1992) and represents the relationship between the means objectives, fundamental
objectives and project measure of the project, highlighting the dependence of objectives on each other and
the measures that are associated with them. This provides an additional aid to stakeholders and project
managers in particular to analyse achievement of objectives, through-out the project cycle. Depending, on
the project context the hierarchy of objectives may change (i.e. a fundamental objective may be a means
objectives or vice-versa in another context). For example, cost containment objective may be a fundamental
for a cost reduction project yet it may be only a means to another objective in another setting. This further
highlights the benefit of this approach through its focus on understanding the rationale or importance of the
objective within each decision context, which are solely guided by a restrictive set of objectives.
www.ejise.com
143
ISSN 1566-6379
Figure 3: An illustration of the objective-measure network
5. Towards tighter coupling with practice

Whetten (1985) provided researchers with some important considerations in the theory development
process. He explained that in the theory development process it is important to clearly state the what, why
and how of the research contribution. The what includes the concepts that are important in the research,
how describes the casual relationships between the concepts or constructs, while why explains the
justification for the research. Essential in the concept development is the need for comprehensiveness (are
all relevant factors included) and parsimony (are there factors that add little value to our understanding) in
the explication of theoretical models. The when, where and who are also important as it these provide the
temporal and contextual factors, the boundaries of generalisability or limitations of the research and the
intended audience of the research. He further explained that this facilitates better understanding on what is
going on through an appreciation of where and when it is happening. Such an approach for theory
development provides tighter coupling of knowledge claims and research solutions.
Against this background, eight (8) experts and project stakeholders in the PM and IS fields were asked to
assess the POMM in order to establish its suitability, usefulness and practicability for practice, including
comparing with the current standard practice of measuring IS projects, which these are sometimes poorly
defined. This exploratory investigation is aimed at achieving tighter coupling with practice through an
interactive design cycle in which practitioners feedback are used to help refine the proposed model. This
approach sets the stage for an explication of the principles of POMM through the case study illustration.
5.1 A preliminary evaluation of POMM

Two (2) rounds of interviews with IS PM experts were conducted to assess the practicability, usefulness and
limitations of the POMM for practice. The interviews also included discussions based on issues and
responses raised throughout the sessions. The first round was used to gather the perspectives of the
practitioners, and the second interview round was used to clarify points and gather more supporting
evidence for the arguments provided by the participants. The stakeholders played several different roles
within their organisations (Table 2) and had shared experiences in managing and assessing IS projects.
More importantly, these individuals have an important stake in finding alternative methods outside their
current practice of adopting the standard methodology. This factor was the key motivation for their selection
in this empirical evaluation combined with their availability to critically discuss the issues. They were asked to
critique the model and offer their perspectives and interpretation through a questionnaire developed based
on the objectives of our investigation and underlined by the principles highlighted by Whetten (1985).
www.ejise.com
144
Table 2: Stakeholders demographic data

Role (# of individuals)
Project Consultant (2)
Project/Programme Manager
(4)
Executive Management (2)
Years of
Experience
10-20+
Industry
5 15+
Information Systems, Software Development,

Telecommunication
Financial Services, Telecommunications
15+
Banking, Information Services
5.1.1 Findings & discussion

The interviews took a structured path; the model was distributed to them initially, during the meetings it was
explained and supported with visual aids. Subsequently, they were allowed to asked questions to clarify their
understanding of the model and its objectives. Upon confirmation that their understanding was consistent
with the models intent they were asked to critically assess it.
5.2 Expected benefits

The model takes into consideration a wider set of objectives through the attention to the breadth of key
stakeholders responded an executive manager. Another suggested that its use may provide some insights
into IS projects that have failed by the traditional approach yet successful upon implementation and use, or
vice-versa. The interviewee went on to claim that few [of their] projects completely fail even they deliver
some value to the organisation. This underscores the need to have a method to help identify the values
important to these stakeholders and the organisation as a whole, and provide a process through which these
can be translated into quantifiable measures. Some of the project managers expressed that the POMMs
philosophy was aligned to their current thinking as their experiences have shown that a project outcome may
be successful in terms of increased revenue or market share, despite it being over budget or late for
example. In addition, the model provides beneficial results through the improved identification of project
objectives, strengthening of the project design process and the evaluation mechanisms and process. The
case was also made for improved decision-making through the analysis of factors that may not have been
previously considered or identified. An example was cited of a project that has been implemented but has
had severe environmental challenges; it was proposed that with the use of the POMM approach some of
these issues may have been brought to the fore, and additional stakeholders may have been taken into
consideration. Improved stakeholder management was also highlighted as this could be enhanced through
the open communication and the perception that their perspectives are deemed important in the elicitation of
key project objectives.
This [approach] could be use when we are developing our business case. An executive noted that with
such an approach the project justification may become clearer and thus created a better opportunity for
stakeholders buy-in while possibly identifying possible trade-offs and help mitigate some of the potential
project risks. This is an important value proposition as it highlights the value of the models flexibility in
enhancing project justification, design and the evaluation of the project itself. The detailed investigation of the
project stakeholder objectives through the POMM approach allows improved analysis at the onset of the
project selection process which can help eliminate or reduce unknowns.
5.3 Factors impacting successful adoption

Some of the essential factors that may influence the models successful adoption in practice were similar to
the discussed on the extant literature on critical success factors (e.g. Pinto and Slevin, 1987, Morris and
Hough, 1987). The participants highlighted executive support and buy-in and stakeholder commitment to the
process as important consideration in the models success in their respective organisations. Other key
factors suggested were the level of awareness of the techniques of the POMM and conformance to its
standards by stakeholders. Hence it is anticipated that the learning curve may be steep during early
adoption.
A project manager suggested that organisational buy-in may be difficult as project managers and
organisations may find it difficult to move from the standards e.g. PMI Body of Knowledge (PMI, 2004) It is
difficult to walk away from the [project management] bible. This was reinforced by another expert who
expressed reluctance in adopting it in their project because of the continued use of the established practice
of conformance to time, cost of specifications. I am more comfortable with a method that is tried and
proven. Notably, this was despite agreement of its practical benefits. This finding reinforces the prominence
www.ejise.com
145
ISSN 1566-6379
of standard bodies methodology and its strong influence on practice (Morris et al., 2006). Additionally, this
phenomenon may be further explained by several technology adoption theories, particularly intuitionalism
(c.f. DiMaggio and Powell, 1983).
Some of the participants saw stakeholders as a restrictive set, primarily focusing on the most influential (i.e.
executive management) without much initial considerations for wider project context such as the clients,
external consultants and other third parties. We propose that more formal approaches to identify project
stakeholders are needed for the critical process (Sharp et al., 1999).
The main concerns raised were the issue of prioritization of objectives and the models adaptability to small
organisations. The interviewees noted despite best efforts some objectives may be more critical than others
and it is essential not to be swayed solely by the most influential project stakeholder. Keeney (1992) stated
that one the principal benefits of the VFT technique was avoiding conflicting decisions as the rationale for the
objectives become known. This point becomes more apparent in the illustration of the POMM, because in
asking the question of values and why each are important helps to classify (and prioritize) the objectives into
means and fundamental objectives wherein all the fundamental objectives will be top priority. Alternatively,
multi-criterion decision techniques such as the Analytic Hierarchy Process (AHP) is useful in prioritizing
diverse perspectives within a particular context (Saaty, 1990) since it can help resolve complex problems
involving high stakes and diverse perceptions (Bhushan and Rai, 2004). The issue of adaptability was raised
primarily because of the assumption that smaller organisations may not have the requisite resources or skillsets to use the model effectively. This is indeed an important consideration; however a key consideration of
the POMM is its applicability to different contexts, organisations or project sizes. The model is accompanied
by simple steps that may used by any individual or organisation. As noted earlier a challenge may be the
initial learning curve, however the potential benefit of greater project clarity and justification may outweigh
that. Additionally, where organisations notice a knowledge gap, this can be resolved through the hiring of
qualified consultants or investing in training programmes for staff.
6. A practical illustration of POMM

A single case study was used to illustrate the concepts and contributions of POMM. The case study
approach is recommended for the in-dept study of an object or model in the business environment (Hevner
et al., 2004). The case study approach embodies an account of past or current phenomenon drawn from
multiple sources of evidence (Leonard-Barton, 1990, Yin, 2003) and involves an empirical inquiry that
investigates a contemporary phenomenon within its real-life context; when the boundaries between
phenomenon and context are not clearly evident; (Yin, 2003). Several researchers advocate this
methodology for investigating real-life events, including organisational and managerial processes
(Orlikowski, 2000, Darke et al., 1998, Yin, 2003) and allows the researcher to retain the meaningful and
holistic characteristics of real-life events (Yin, 2003). Additionally, the case study method is applicable
because of its suitability in providing enhanced understanding to organisational contexts and IT-related
innovations (Darke et al., 1998). The illustration of the POMM makes the case for its usefulness in providing
an alternative evaluation technique for IS project outcome.
The study considers a IS graduate programme development project at a Caribbean university, which
includes activities such as the programme design and students thesis development process and review of
the thesis outcome. This activity was chosen for several reasons: 1) it clearly makes the case that the
traditional method of assessing project outcome is insufficient because the success of the graduate
programme involves more that conformance to time, cost and specifications as determined by the University;
and 2) there is a great possibility that each set of stakeholders have their unique set of objectives combined
with common set of the programmes project objectives. Therefore, the steps of POMM may be clearly seen
through this view.
Identify Key Stakeholders. The thesis development process involves numerous stakeholders. The task
involved considering the roles and actors that may be affected by the programmes evolvement. This
process was achieved through documents review and elicitation through discussions. Documents relating to
the programme were reviewed to identify key roles that are involved in the programme, and specific
stakeholders were asked to consider the process and the history of the programme to help identify groups or
individuals that played a part in its development and management. The stakeholder entities were later
grouped into personal, school community, academic and business communities were identified. The principal
stakeholders therefore included advisors, administrators, internal and external examiners, executive
sponsor, corporate sponsors or organisations. The stakeholder groups interviewed were students, advisor,
www.ejise.com
146
academic director, internal examiner, executive sponsor and an organisational representative. This was
done to obtain a fair representation of the stakeholder groups and their availability in being interviewed.
Elicit Project Values and Objectives. The first stage involved interacting with stakeholders in an open
forum to garner the essential elements that were important to them in the dissertation programme. Using the
guideline set out by Keeney (1996), the participants were asked questions such as what do you value or
want from the dissertation project, and what are your ultimate objectives of the dissertation project. Upon
completion of all interviews, the notes were reviewed and a complete list of all the objectives identified was
prepared. The product of this was a list of 46 project objectives. The second stage involved reviewing the
objectives and converting them into common or consistent form as several of the objectives represented the
same thing but were stated differently. This resulted in 26 objectives (see figure 3) which laid the foundation
for the next step in the process.
Determine the Relationship between Objectives. Each participant was asked why each of their identified
objectives was important. During this exercise it was discovered that this approach may result in additional
objectives as the participants get an opportunity to elucidate more on what they value. For example, in
discussing the importance of a high quality dissertation development process, the advisor noted that
ignorance of stakeholders (e.g. student, committee role) may lead to conflicts and they did not want any
occurrence of unfairness or lack of objectivity to be experienced. Therefore along with an objective for clarity
among stakeholders role, the issue of objectivity in the review process was also highlighted. Also, as
expected the exercise resulted in identifying the relationships among the objectives. For example, the
student identified the need to improve research skills or enhance their understanding of research
methodologies, however in asking why these were important in addition to completing the dissertation, the
value of growth in the IS community or to produce relevant research were identified. Nine (9) fundamental
objectives were identified and the list of all the objectives and relationships were converted to the project
means-end network (see figure 4).
Determine the Priorities of the Project Fundamental Objectives. A simple rating method was used based
on the decision context. The stakeholders were asked to rate the importance of the fundamental project
objectives in determining a successful IS PhD programme. The level of importance ranged from 1-9, based
on the number of project fundamental objectives identified, and was defined as follows: High = 7-9, Medium
= 4-6, and Low = 1-3.
Each stakeholder was asked to explain their decisions in determining the ratings and an average score was
produced as the final perspectives on priority of the objectives. Interestingly, none of the objectives received
a rating below five which implies that the stakeholders valued each of the fundamental objectives and likertlike scale with closer range could be used. The higher priority objectives included: exhibit intellectual
independence, competence in the research process IS content knowledge and a formal process for IS PhD
programme development, see fig. 3.
Table 3: Fundamental project objectives & priorities
List of Project Fundamental Objectives
Average
Rating
Weight
Exhibit intellectual independence
9.00
0.124
Develop competence in research process
9.00
0.124
Possess doctoral-level IS content knowledge
8.67
0.119
Develop/enhance a formalised process of IS dissertation management (inclusive of development,

review, etc)
8.33
0.115
Conform to ethical standards of the profession
8.33
0.115
Maximize the number of completed dissertation
7.67
0.106
Create a legacy of research
7.33
0.101
Dissertation effort must provide practical and relevant research to the Caribbean
7.33
0.101
Maximize opportunity to become established in the research community
7.00
0.096
www.ejise.com
147
ISSN 1566-6379
Review Means-End Network. A review of the network was performed to ensure completeness of the
objectives and accuracy and completeness of the relationships. The main questions asked during this
process were: did it make sense and did it accurately reflect the views of the stakeholders. Minor refinements
were performed to address missing links and consistencies among the project objectives.
Elicit Project Measures. The participants are asked to explain how they would measure the achievement of
the stated objectives, those they provided initially and the full composite of project objectives. The GQM
approach was used to analyse the data, elicit and identify measures associated with the project objectives.
There are two strategies that may be employed in obtaining the measures by identifying the measures linked
to the project means objectives or linked to the project fundamental objectives. This examination analysed all
the project objectives to identify suitable measures. In eliciting the measures, the purpose of the objective is
first identified, followed by the its issue, object or aim of the objective, and viewpoint through which the
objective is seen (Basili et al., 1994). The set of questions are then developed from which the measures of
viewing the objectives derived and the process was repeated for all the objectives. For example, the
objective of Possess Doctoral-Level IS Content Knowledge is analysed by identifying purpose of the
objective (i.e. attain), the issue (i.e. possession), the object of the objectives (i.e. doctoral-level IS content
knowledge), and the viewpoints through which the objectives may be viewed (i.e. student, advisor).
Questions include (1) What are the characteristics that indicate or reflect doctoral-level IS content
knowledge? and (2) Is formal assessments of students content knowledge performed? Therefore metrics
include (1) Score on comprehensive exam, (2) GPA on dissertation course work, and (3) Level of
comprehensiveness of literature review section.
The approach allows for objective and subjective ratings of the objectives (Basili et al., 1994), so an objective
measure would be the exam results and a subjective measure of an advisor would be the rating on the level
of comprehensiveness of the dissertation or sections of it. It was observed that the exercise of identifying
metrics was also made easier with the use of the VFT technique. It became evident that the VFT provided a
clearer vision of what the stakeholders needed (richer objectives), and it was observed that the act of
developing questions and finding associated metrics was made easier with the aid of the project means-end
network. The project measures are later summarized in the Fundamental Objectives-Measures network, a
product of the POMM displayed in Table 4.
The reader may observe that each objective has multiple measures, and that measures appear to have
different scales. Through the use of appropriate transformation rules, scores for all measures could be
converted to a uniform scale (e.g. High = 7-9, Medium = 4-6, and Low = 1-3). For a given fundamental
objective, the scores of its associated measures on the uniform scale would be synthesized using some
method (e.g. averaging) resulting on an overall score for the fundamental objective.
www.ejise.com
148
Figure 4: Project means-ends network - IS Thesis Programme Project
www.ejise.com
149
ISSN 1566-6379
Table 4: Project fundamental objectives-measures network

Project Fundamental Objectives
Develop competence in research process
Measures
Subject course grades
Rating on dissertation research development
Possession of doctoral-level IS content

knowledge
Course work grades

Comprehensive exam results
Rating on completed dissertation
Rating on Literature Review section of dissertation
Possess intellectual independence
# of independent research activities

# of independent publications
Ratio of independent activities/publications to collaborative
activities/publications
Rating on level of intellectual growth during programme
Detailed procedures on review and assessment process

Detailed procedures on accountabilities, structure, how the
programme works
Results of benchmark against other universities
Results of independent audit of programme
Conform to ethical standards of the profession
Rating of the standard of the completed dissertation

Rating of conformance to standards set-out by IS profession
Maximize dissertation completion
# of completed dissertation
Ratio of completed dissertation to enrolled students
Average latency of completion
Provide strong legacy of research
# of graduates
Rating of cadre of students/graduates
Creation of facilities that promote Caribbean research/IS
research (e.g. a conference)
Produce practical & relevant research
# of publications
# of practitioner-based publications
# of research adopted in practice
Rating on the level of interest shown by practice (e.g.
business community)
Establish self in research community
# of publications
# of citations of published works
# of consultations and/or requests
# of invitations (reviews, research community service)
Develop/enhance a formalized dissertation

mgmt process
Review Objective-Measure Network. The participants were later asked to review the means-ends network
to determine whether 1) it represented what they wished to convey 2) there were any missing relationships,
objectives or measures, and 3) it was an understandable tool that would be useful to them in generating
richer project objectives. The stakeholders conveyed that the model and approach was useful as it highlights
the objectives, relationships and measures that may be used to assess the achievement of same.
Monitor Project Objectives. The performance of the project is monitored against the achievement of the
objectives based on the measures identified. Equipped with the set of objectives and measures, the
university (e.g. programme coordinator) may assess the progress or achievement of the respective
objectives and determine whether any refinement or corrective actions are needed throughout the remainder
of the project. For instance, in monitoring intellectual independence of students, the number of independent
publications would be evaluated or ratio of independent and collaborative activities. These may be further
compared against agreed baselines that are included in the programmes policies and procedures (objective
formalization of programme). If performance is below expected standards, corrective measures such as
research workshops may be conducted and independent research activities reevaluated. Evidently, as seen
by the diverse objectives some may be achieved within the short term and others over a longer term, for
example, course grades and procedures can be evaluated during the life of the programme, the number of
graduates at the first cycle and number of publications per student over a longer period. This is aligned with
www.ejise.com
150
trends of including the product life cycle assessments as part of the evaluation of outcome of the project
(Nelson, 2005).
Determine the Project Outcome. In the determination of the project objectives and measures practitioners
now need a strategy to define the final outcome of the project in terms of success or failure or anything in
between. Such an approach includes an assessment of the achievement of all the fundamental objectives
and the use of a matrix as defined in Table 5 to define the achievement of an outcome. Table 5 includes a
set of decision rules that could be used to assess the overall project outcome. For example, achievement of
all fundamental objectives equates to a highly successful project, achievement of some of the fundamental
objectives combined with all the other equates to a partially successful project outcome. Achievement of
none of the fundamental objectives would equate to failure. All of the possible combinations are not shown,
but rather an illustration of the decision rules that may guide practitioners in the determination of a clear
project outcome.
Table 5: An example of project outcome determination
High priority
Medium priority
fundamental objectives
fundamental objectives
All
All
All
Most
All
Most
All
Some
Most
All
Most
Most
Most
Most
Most
Some
Most
None
Some
All
Some
Most
Some
Some
Some
Some
None
All
None
None
*** All other Combinations ***
Low priority fundamental

objectives
All
All
Most
All
All
All
Most
Some
None
All
Most
All
Some
All
All
Project Outcome Results

Highly successful
Highly successful
Highly successful
Highly successful
Mostly successful
Mostly successful
Mostly successful
Mostly successful
Partially successful
Challenged
Challenged
Challenged
Failed
Challenged or Failed
Alternately the overall assessment of the project could be determined by using the weighting model as
determined in the Determine the Priorities of the Project Fundamental Objectives step (see Table 3) and
scores for the fundamental objectives as were determined in the Elicit Project Measures step. Other options
include multi-criteria decision analysis (MCDA) methods such as AHP or Electre, which are useful strategies
for assisting in the determination of the final outcome of the project. These methods have been cited for
being most reliable or user-friendly based on a review of the literature. Hence, there is improved likelihood
they can be easily adopted in this environment.
7. Concluding remarks
The POMM is proposed as a tool that can aid in the evaluation process of IS projects through the elicitation,
development and alignment of project objectives and measures reflective of the values of the key project
stakeholders. An exploratory assessment of the strengths and implications of the proposed model by a team
of IS PM experts was performed. They concluded that the model can provide value to organisations through
the identification of stronger set of project objectives, or even at the project selection/justification phase
through improved project design, and is an improvement of the current standard methodology practiced in
their organisations. The case study method was used to illustrate the procedures and concepts of POMM
using an IS thesis programme. While this case may not have fallen into the typical business domain, we
proffer that IS plays a role in every sector and the model can be applied within any domain seeking to
evaluate their IS projects.
This research represents the first in a series of studies to extend the debate on perspectives of IS project
evaluation tools. It is anticipated that this study will help project practitioner rethink their evaluation methods,
and focus on a more complete approach that engenders stronger project design and improved accounting of
project delivery. We argue that this will aid stakeholder satisfaction. Future research directions involve
enhancing the generalisability of the study through further examination and refinement of the POMM and its
application in diverse IS project case settings.
www.ejise.com
151
ISSN 1566-6379
References
Agarwal, N. and Rathod, U. (2006) Defining 'success' for software projects: An exploratory revelation, International
Journal of Project Management, Vol. 24, No. 4, pp 358-370.
APM (2006) The APM body of knowledge, The Association of Project Management
Atkinson, R. (1999) Project management: Cost, time and quality, two best guesses and a phenomenon, its time to accept
other success criteria, International Journal of Project Management, Vol. 17, No. 6, pp 337-342.
Atkinson, R., Crawford, L. and Ward, S. (2006) Fundamental uncertainties in projects and the scope of project
management, International Journal of Project Management, Vol. 24, pp 687698.
Basili, V. R., Caldiera, G. and Rombach, H. D. (1994) The goal question metric approach, Encyclopedia of Software
Engineering,
Basili, V. R. and Weiss, D. M. (1984) A methodology for collecting valid software engineering data, IEEE Transactions on
Software Engineering, Vol. 10, No. 6, pp 728-738.
Belassi, W. and Tukel, O. I. (1996) A new framework for determining critical success/failure factors in projects,
International Journal of Project Management, Vol. 14, No. 3, pp 141-151.
Bhushan, N. and Rai, K. (2004) Strategic decision making: Applying the analytic hierarchy process, Springer-Verlag,
London
Bourne, L. and Walker, D. H. T. (2006) Using a visualising tool to study stakeholder influence - two Australian examples,
Project Management Journal, Vol. 37, No. 1, pp 5-21.
Brynjolfsson, E. (1993) The productivity paradox of information technology, Communications of the ACM, Vol. 36, No. 12,
pp 67-77.
Cicmil, S., Williams, T., Thomas, J. and Hodgson, D. (2006) Rethinking project management: Researching the actuality
of projects, International Journal of Project Management, Vol. 24, pp 675686.
Cleland, D. I. (1986) Project stakeholder management, Project Management Journal, Vol. 17, No. 4, pp 3644.
Courtney, J. F. (2001) Decision making and knowledge management in inquiring organizations: Toward a new decisionmaking paradigm for DSS, Decision Support Systems, Vol. 31, pp 1738.
Crawford, L., Morris, P., Thomas, J. and Winter, M. (2006) Practitioner development: From trained technicians to
reflective practitioners, International Journal of Project Management, Vol. 24, No. pp 722733.
Darke, P., Shanks, G. and Broadbent, M. (1998) Successfully completing case study research: Combining rigor,
relevance and pragmatism, Information Systems Journal, Vol. 8, No. pp 273-289.
DiMaggio, P. J. and Powell, W. W. (1983) The iron cage revisited: Institutional isomorphism and collective rationality in
organizational fields, American Sociological Review, Vol. 48, No. 2, pp 147-160.
Drevin, L., Kruger, H. A. and Steyn, T. (2007) Value-focused assessment of ICT security awareness in an academic
environment, Computers & Security, Vol. 26, No. 1, pp 36-43.
Engwall, M. (2003) No project is an island: Linking projects to history and context, Research Policy, Vol. 32, No. pp 789808.
Fitzgerald, G. (1998) Evaluating information systems project: A multidimensional approach, Journal of Information
Technology, Vol. 13, No. 1, pp 15-27.
Fortune, J. and White, D. (2006) Framing of project critical success factors by a systems model, International Journal of
Project Management, Vol. 24, No. 1, pp 53-65.
Hevner, A. R., March, S. T., Park, J. and Ram, S. (2004) Design science in information systems research, MIS Quarterly,
Vol. 28, No. 1, pp 75-105.
Irani, Z. (2002) Information systems evaluation: Navigating through the problem domain, Information & Management,
Vol. 41, No. 1, pp 11-24.
Jugdev, K. and Muller, R. (2005) A retrospective look at our evolving understanding of project success, Project
Management Journal, Vol. 36, No. 4, pp 19-31.
Kaplan, R. S. and Norton, D. P. (1992) The balanced scorecard: Measures that drive performance, Harvard Business
Review, Vol. 70, No. 1, pp 71-79.
Keeney, R. L. (1992) Value-focused thinking: A path to creative decision-making, Harvard University Press. pp. 432
Keeney, R. L. (1996) Value-focused thinking: Identifying decision opportunities and creating alternatives, European
Journal of Operational Research, Vol. 92, No. pp 537-549.
KPMG (2005) Global IT Project Management Survey, KPMG Information Risk Management, pp. 1-35.
Kumar, R. L. (2003) Understanding the value of information technology enabled responsiveness, Electronic Journal of
Information Systems Evaluation, Vol. 1, No. 1, pp.
Leonard-Barton, D. (1990) A dual methodology for case studies. Synergistic use of longitudinal single site with replicated
multiple sites, Organisation Science, Vol. 1, No. 1, pp 248-266.
Melville, N., Kraemer, K. and Gurbaxani, V. (2004) Review: Information technology and organisational performance: An
integrative model of it business value, MIS Quarterly, Vol. 28, No. 2, pp 283-322.
Merriam Webster Dictionary (2008) [online] http://www.merriam-webster.com/
Morris, P. W. G., Crawford, L., Hodgson, D., Shepherd, M. M. and Thomas, J. (2006) Exploring the role of formal bodies
of knowledge in defining a profession the case of project management, International Journal of Project
Management, Vol. 24, No. pp 710721.
Morris, P. W. G. and Hough, G. H. (1987) The anatomy of major projects: A study of the reality of project management,
John Wiley & Sons. pp. 338
Nelson, R. R. (2005) Project retrospectives: Evaluating project success, failure and everything in between, MIS Quarterly
Executive, Vol. 4, No. 3, pp 361-372.
www.ejise.com
152
Orlikowski, W. J. (2000) Using technology and constituting structures: A practice lens for studying technology in
organisations, Organization Science, Vol. 11, No. 4, pp 404-428.
Pinto, J. K. and Slevin, D. P. (1987) Critical success factors in successful project implementation, IEEE Transactions
Engineering Management, Vol. 34, No. 1, pp 22-27.
PMI (2004) A guide to project management body of knowledge, Project Management Institute
Saaty, T. L. (1990) How to make a decision: The analytic hierarchy process, European journal of operational research,
Vol. 48, No. 1, pp 9-26.
Sharp, H., Finkelstein, A. and Galal, G. (1999) Appeared in 10th International Workshop on Database & Expert Systems
Applications, Vol No. pp 5.
Sheng, H., Nah, F. F.-H. and Siau, K. (2005) Strategic implications of mobile technology: A case study using valuefocused thinking, Journal of Strategic Information Systems, Vol. 14, No. pp 269290.
Shenhar, A. J., Dvir, D., Levy, O. and Maltz, A. C. (2001) Project success: A multi-dimensional strategic perspective,
Long Range Planning, Vol. 34, No. pp 699-725.
Shenhar, A. J. and Levy, O. (1997) Mapping the dimensions of project success, Project Management Journal, Vol. 28,
No. 2, pp 5-13.
Shenhar, A. J., Tisler, A., Dvir, D., Lipostevsky, S. and Lechler, T. (2002) Refining the search for project success factors:
A multivariate typographical approach, R & D Management, Vol. 32, No. 2, pp 111-126.
Smithson, S. and Hirschheim, R. (1998) Analysing information systems evaluation: Another look at an old problem,
European Journal of Information Systems, Vol. 7, No. 3, pp 158-174.
Solingen, R. v. and Berghout, E. (1999) The goal/question/metric method, a practical method for quality improvement of
software development, McGraw-Hill
Wateridge, J. (1998) How can is/it projects be measured for success? International Journal of Project Management, Vol.
16, No. 1, pp 59-63.
Whetten, D. A. (1985) What constitutes a theoretical contribution? Academy of Management Review, Vol. 14, No. 4, pp
490-495.
White, D. and Fortune, J. (2002) Current practice in project management - an empirical study, International Journal of
Project Management, Vol. 20, pp 1-11.
Winter, M., Smith, C., Cooke-Davies, T. and Cicmil, S. (2006a) The importance of process in rethinking project
management: The story of a UK government-funded research network, International Journal of Project Management,
Vol. 24, No. pp 650662.
Winter, M., Smith, C., Morris, P. and Cicmil, S. (2006b) Directions for future research in project management: The main
findings of a UK government-funded research network, International Journal of Project Management, Vol. 24, No. pp
638649.
Yin, R. K. (2003) Case study research: Designs and methods, Sage Publications
www.ejise.com
153
ISSN 1566-6379
www.ejise.com
154
Economic Desirability and Traceability of Complex Products

Mordechai Ben-Menachem and Ilanit Gavious
Ben-Gurion University, Beer-Sheva, Israel
madaril@bgu.ac.il
quality@acm.org
Abstract: The real values and benefits of Information Technologies are difficult to quantify and frequently even to identify
accurately. Existing financial models such as Net Present Value have proven insufficient for complex products, for longterm corporate goals. IS projects and software-rich products are decided upon while ignoring critical financial aspects, as
the distance between the corporate product vision and the reality that engineers see may be very large. This paper maps
between economics vis--vis IS-based product management via an inter-disciplinary approach, looking at the needs and
exigencies of corporate management, IS project, products and software engineering.
The basis for the article is a discussion of the difficulties in evaluation of the economic desirability of complex, softwarerich products. It presents a dynamic corporate-level model for economic profit evaluation designed to deal with the
unique characteristics of such products, over many variants and versions, and the entire lifecycle. Given the extreme
uncertainty of costs, benefits, risks and timeframes projections, the model facilitates real time reporting via an information
system designed for management of Products, Portfolios and Projects. Whereas existing project management
techniques such as Earned Value Management provide a general basis for managing project level activity, our model
provides a longer-term view to assess economic affects of corporate strategies over time. This is provided by a dynamic,
Management Information System based aggregation of all product information, over an entire product lifecycle, with the
objective to provide a knowledge base for corporate dynamic decision-making. Concomitantly, the model fulfils
Sarbanes-Oxley Act of 2002 requirements for management assertion traceability of valid and accurate measures.
These aspects co-joined, from Sarbanes-Oxley, back through multiple products, over myriad versions, and through
automated requirements, design and testing tools, all combine to form an auditable management feedback loop that can
be leveraged at multiple corporate management levels. The paper represents a significant step towards quality product
decision-making via a model that is meaningful, while also useful as it is leveraged through an automated tool set.
Keywords: economic profit, information systems, IS management, IS evaluation, product management, project
management, traceability
1. Introduction
In the modern economy management seeks enhanced measurement capabilities of economic benefits
provided by complex, software-rich, products. This economic benefit is usually derived from costs savings,
revenue increase and/or enhanced resource-usage efficiency. Project managers are required to provide
corporate management with a supporting environment for quality decision-making; while products compete
for resources within an environment of limited resources.
Extant research examines issues of project efficiency maximization. For example, a recent study by Serich
(2005) shows how the combination of Concurrent Engineering with Prototyping enhances project efficiency.
However, what is still lacking is the multi-dimensional facet of the economic value of the product to the
organization. Notably, economic value cannot be limited to the period in a product/system lifecycle devoted
to initial development, but needs to be viewed in a longer context.
Corporate environments are characterised by multi-layered product portfolios; wherein each separate
product is created, maintained, enhanced and/or expanded via the facility of a project. Basic data concerning
these processes begin from the project level but for corporate decision-making, need to be aggregated to
product, and frequently portfolio, level. This demands a dynamic decision-making instrument.
One of the most common financial tools used for investment decision making is Net Present Value (NPV).
Literature referring to economic modelling for software-rich products is very sparse; whether in computing
literature, finance literature or management literature (e.g., Turnbull, 2003; Greiner, 2003; Armour, 2005).
Boehms (1981) well-known book Software Engineering Economics deals mostly with software engineering
aspects, and not the economic or management aspects. Tockey (2005) is the first book to address this area
specifically. His orientation is for the software professional, and is based on an expectation that the usual
software practitioner is unfamiliar with economic aspects. This gap between the two professions creates an
economic reality in which Information Systems (IS) projects, the basis of every modern organization's
economic existence, are being decided upon while ignoring many critical financial aspects.
ISSN 1566-6379
155
Ben-Menachem, M. and Gavious, I. Economic Desirability and Traceability of Complex Products. The Electronic
Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 155 - 166, available online at www.ejise.com
This article presents a dynamic model for economic evaluation of the desirability of complex products.
Throughout this article, the term complex products refers to software-rich products, containing in their
lifecycle, multiple versions (changes over time) and multiple variants (changes in use). The model is based
on analysis of critical differences between complex and more traditional products. These differences are
found in costs and expected benefits from the product as well as associated risks and timeframes. Our
model applies these differences as measurable information, collected from real objects and events, by an
information system.
Existing project management techniques, such as Earned Value Management (EVM), provide a general
basis for managing project level activity. A project management approach is sufficient in an environment of
incremental development or a one up project. Our model provides a longer-term view to assess economic
affects of corporate strategies over time. This is provided by dynamic, Management Information System
(MIS) based aggregation of all product information, over an entire product lifecycle. The objective is to
provide a knowledge base for corporate dynamic decision-making. Concomitantly, our model supplies
business goals traceability via specific procedures. This is consistent with the Sarbanes-Oxley Act of 2002
(Section 203) requirement for enterprises to implement traceability and change-management, from
management assertions through components.
The paper is organized as follows. Section 2 discusses principles of economic evaluation of complex
products, focusing on the traps in economic modelling. Section 3 presents a multi-dimensional analysis of
costs and benefits, with incorporation of risks and timeframes for complex products. Thus providing basis for
an economic profit model adapted for the unique characteristics of complex products. Section 4 presents our
dynamic model for evaluation of complex products' economic desirability, designed for constant updating via
a functioning Management Information System. This dynamism is shown to be crucial in such products
because of the extreme uncertainty embedded in projections of costs, benefits, risks and timeframes.
Section 5 contains summary and concluding remarks.
2. Economic evaluation of complex products desirability

Managers generally base business decisions on evaluation of costs and expenses versus revenues, or other
potential benefits. Management needs to perceive which resource investments may produce an optimal
result, which should be economically expressible. One possible way to represent this conceived optimum
might be via estimation of the expected return on a particular investment, as compared to alternative
investments though a specific project may be chosen even though economic models may not show it the
optimal investment (for instance from considerations such as market competitiveness, customer services,
quality or other not-readily-quantifiable corporate needs). This is made much more difficult in a software-rich
project. Such projects should be measured using standards of performance similar to traditional, tangible
asset-based projects. However, while they are expected to stand under the same magnifying glass of
economic profit measures, they are not expected to produce the same kinds of profit as benefits may
frequently be qualitative (e.g., customer satisfaction). Furthermore, looking at the entire lifecycle of a
complex product, rather than a single project, complicates the decision-making process.
The use of a financial model to measure product desirability is valuable. A financial model supplies a clear
framework for forecasting costs slated for an investment and evaluating its expected benefits. It enforces a
1
well-designed implementation plan from all stakeholders, with viewpoint commonality. A model helps
organizations understand and concur on expected results. At the least, such a model is a common tool to
show all stakeholders, from the lowest level to senior management that an investment has economic value.
However, a significant trap in economic modelling is the inherent inaccuracies embedded in forecasting
future cash flows and cost of capital over investment lifecycle. This is made much more problematic in
software-rich products as development times may be quite long. Moreover, economic value cannot be limited
to the period in a lifecycle devoted to initial development, but also needs to account for on-going system
evolution (ASD C3I 2002; Ben-Menachem & Marliss 1997; BPM Forum 2004). The temporal gap between
the projects initial requirements analysis and the concomitant requirements as the system or product
matures, then serves to forcibly grow the aforementioned inaccuracy. This is more than just requirements
drift, which tends to infer changing of agreed upon requirements while in development, and includes new
1
As defined by IEEE Standard 1471: [3.1.9] view: a representation of a whole system from the perspective of a
related set of concerns. [3.1.10] viewpoint: a specification of the conventions for constructing and using a view. A
viewpoint acts as a pattern or template from which to develop individual views by establishing the purposes and
audience for a view and the techniques for its creation and analysis.
www.ejise.com
156
requirements resulting from the learning processes inherent to development. The temporal issue has
additional effects, as while the product and technology evolve, other effects occur, such as changes in the
business environment.
Another trap is that quantifiable methods for estimating intangible results are a major challenge; while
frequently, these intangibles may be the most significant (for example, enhanced Information Retrieval
ease/speed or an ability to make higher-quality business decisions). Despite these weaknesses, financial
modelling is usable even in complex products, provided the model and the modelling techniques are adapted
to the product. We make this adaptation in a multi-dimensional costs and benefits analysis, with the
incorporation of risks and timeframes, for complex products.
3. Multi-dimensional costs and benefits analysis of complex products

Our model for economic evaluation of complex products' desirability is based upon the traditional NPV
method of evaluating the desirability of investments. This section shows the basis for our model and critical
points of difference between it and NPV. These differences are in the areas of costs and expected benefits
from the product as well as associated risks and timeframes.
3.1 Costs analysis

A deep understanding of costs is critical for good product planning. The Information Technology industry is
rife with reports concerning projects failure rates (Clancy, 1998; Taylor, 2001). Enhanced understanding of
costs involved in a project will enhance product managers ability to effectively plan products, as well as
enhance abilities to react to changes in real time. Studies that deal with this issue (e.g., Boehm, 1997) tend
to address initial development exclusively, with little or no discussion of on-going costs.
Complex products' characteristic costs divide into those that are typically predictable and those that are
known ex post facto. The former include cost of hardware (equipment of all kinds), communications networks
(infrastructure and connection costs), software licenses, manpower (technical and non-technical personnel,
organic to the organization or external consultants employed for temporary tasks), training (staff and users)
and installation (including data conversion). Although predictable, these costs pose a high degree of
uncertainty, temporal and otherwise. For example, whereas consultants may be hired for exactly the times
needed, employees availabilities are not pre-determinable by project needs (only) but also by the availability
of the needed personnel; that is, the correct and needed personnel must be available, many of whom may
be specialists.
The second type of costs includes aspects of a more complex structure. This complexity causes them
frequently to display unpredictably quantifiable aspects. Software (and withal, software-rich products) differs
from traditional product development projects in four broad areas, Commercial-Off-The-Shelf (COTS) tools
and modules, quality assurance activities, development processes and system uncertainty.
COTS tools or modules: It is very rare to develop an entire bespoke system/product. Almost all, whether
information or reactive systems, include some vendor-acquired modules and some bespoke. In addition,
most work environments (development and evolution) include vendor-developed software tools (these may
be deliverable with the project or non-deliverable, used only for development). Some applications may
contain both types, e.g., reporting tools and acquired processing modules (the concept of module infers that
it becomes an active part of application operation; while a tool is either passive or external). Packaged
modules and tools predetermine a great deal of the design structure of the chosen solution and highly affect
all development and evolution processes used to produce and maintain product suitability. A high quality,
flexible set may reduce both development and evolution costs. Poor choices will cause many unforeseen
and unforeseeable occurrences. Frequently, a great deal of expert and specialized knowledge is needed to
utilize these purchased items effectively. This knowledge may also be acquired and not necessarily with the
purchased product.
Components purchase is common in every project but there is a very significant difference. In traditional
projects one purchases a component with the clear intention that it be fundamentally a Black-Box. In
software, this is almost never the case. Almost all purchased software components are Grey-Boxes where
functionality is acquired and then needs to be studied (and frequently modified) before it can be utilized, or
utilized optimally. This difference is a very large cost component, above and beyond the purchase cost that
must be accounted for and planned for.
www.ejise.com
157
ISSN 1566-6379
Quality Assurance activities:

Large portions of the resources of software-rich projects are used to correct
errors produced by the project (typically, fifty percent of overall expenditures (Ben-Menachem & Marliss,
1997)). As part of, and in parallel with, the quality assurance process, every project contains many
verification and validation processes, both to capture errors and to allow timely correction after discovery.
Generally, software quality is not easily quantifiable (despite many attempts). Notwithstanding best efforts,
every software system contains some unknown quantity of defects. Moreover, defects also occur in parts of
a system that were previously satisfactory, as requirements and environments change and evolve. Many
costs of non-quality can be computed, but only in retrospect.
Software development processes:
Theoretically, discussions of software development process may not
seem significantly different from many other process-oriented industries - for example, the highly published
processes in the automotive industry (e.g., common platforms use). However, when designing an
automobile, the concept is to design it once and then manufacture as many copies as possible (even if the
copies are not identical, they are technologically non-dissimilar; differing usually in superficial aspects such
as colour schemes). Software does not have manufacturing. It is always designed and produced one-up,
then constantly changed and evolved afterwards. Automobiles are also not designed to be updated while in
use, while software must be constantly updated to remain viable (Lehman & Ramil, 2001).
System uncertainty:
A most prevalent attribute of software-rich projects is high uncertainty; degrees of
uncertainty ensure unforeseen costs. Lehman (1990), in his seminal article concerning computer system
uncertainty shows that: In the real world, the outcome of software system operation is inherently uncertain
with the precise area of uncertainty also not knowable. Additionally, Ben-Menachem & Marliss (2004)
showed that this uncertainty principle extends beyond programs to systems and systems-of-systems (e.g.,
ERP) and more than that, is not static and is a natural aspect of the ways that systems evolve. To whit, all
change increases system uncertainty, which grows exponentially as the configuration item view extends
toward systems and systems-of-systems; anomalies propagate, creating unforeseen states, and inherent
system uncertainty explodes.
Unforeseen costs refers to costs which, while the experienced project manager knows that such will occur
and may even be able to guess-timate their approximate quantities, what these will be, when they will occur
and exactly for what will the money actually be spent, is impossible to define ex-ante.
3.2 Benefits analysis

Potential benefits of complex products may be quite broad. Occasionally, the most significant benefits will be
strategic. Strategic and tactical benefits should be differentiated, as the former frequently are not directly
measurable while the latter may be straightforward (see Tockey, 2005). Strategic might be overall change
the company concepts such as ERP implementation, while tactical might be enhancement of a corporate
function (e.g., customer management).
Many systems surprise the producers, bringing benefits other than those originally defined, some of whom
may actually prove of greater value than the original plan. Not all benefits are, or can be, tangibly defined.
For example, if the new system enables more efficient report production, using less manpower and/or less
time, than the direct benefit is clear and can be quantitatively measured. However, indirect benefits may be
more difficult to measure - staff that previously produced this report may now be free to work on enhanced
services. These are not as easily measured.
Additionally, benefits are not actualized at once. Projects may take significant time to implement (three to five
years is not uncommon) while many other factors may affect the corporate environment during the period.
Direct cause-effect analysis is impracticable, as it may be difficult to isolate a projects benefits from other
causes. However, this does not preclude indirect analysis and/or use of simulations and projections. Preproject evaluation of benefits and costs should be based on projection of the (proposed) situation with and
without the project, and not before and after the project. This is to isolate (proposed) changes to the
organization caused by the project (see also, Tockey, 2005, pp. 305-306) and even more so, by the product
that results from the project (or projects) creating and evolving it.
Whereas benefits analysis of an individual project is difficult and complex, it is significantly more so for an
entire product lifecycle, at least as a product may typically be developed via many individual projects over its
lifecycle.
www.ejise.com
158
3.3 Risks of complex products and their effect on the investment

Longstaff, Chittister, Pethia & Haimes,(2000, pp. 45) state: We cannot simultaneously measure the risks
associated with software and information assurance when no protective actions are taken and measure the
efficacy of deploying risk assessment and management on the system because the system has
fundamentally changed. This extends Heisenbergs Uncertainty Principle, to software, and has quite
profound and far-reaching effects on economic modelling of software-rich systems (since the measuring
device has been constructed by the observer, and we have to remember that what we observe, and what we
observe is not nature itself but nature exposed to our method of question. when searching for harmony in
life one must never forget that in the drama of existence we are ourselves both players and spectators.,
Heisenberg, 1991). What seems the most obvious characteristics of software, to both practitioners and
users, are the lack of wear-down and the ease of changing it. Both perceptions are incorrect and are a major
source of risk (Broekman & Notenboom, 2003). The mis-perception results from viewing software via an
inapplicable paradigm.
The past few centuries used an industrial paradigm for filtering worldview. The industrial paradigm assumes
glacial change; so slow and gradual as to be barely recognizable except over relatively long periods of time;
in hindsight. Todays reality differs. We live in a world where the only certainty is change. Changes occur so
quickly that in most situations we must resign ourselves to them without first being able to assess what their
risks or benefits might be (Asuaga, 2001). Changes have levels of risk associated with them (Broekman &
Notenboom, 2003), and as they occur so quickly, how can they be economically modelled, effectively? If
software always has threats, hazards and risks, what are the roles of the project, and project management,
in identifying, preventing, mitigating and controlling them?
Basic comprehension of risks centrality is critical to economic modelling (Armour, 2005). In order to
comprehend the difficulty inherent to the task of risk management and its concomitant affects on economic
modelling, one must understand that this is not limited to the expected that is, to the risks inherent in
original product development and deployment (see section 3.1 System Uncertainty). Brooks (1995) added
to this the inherent instability of computer systems: Fixing a defect has a substantial (20 to 50 percent)
chance of introducing another. From this is derived the concept that after a complexity threshold is
exceeded, fixing one flaw tends to create new flaws.
Risks in complex products exist on at least two levels, product oriented risks and risks associated with the
projects that create and maintain the product. These are separate and not a linked hierarchy of inter-related
risks.
3.4 Timeframes
The timeframes in which a products desirability is evaluated changes as a function of the type of project(s)
utilised, the type of product/system that the project(s) are intended to develop, the products defined (and
undefined) lifecycle and the changeability of the external environment over the time that the product is in
development. Many researchers have shown that software projects almost always take more time than was
initially expected (Brooks, 1995; Clancy, 1998; Taylor, 2001). The projects original definition frequently
changes over time, sometimes without management knowledge. In addition, Verhoef (2002) shows that
project costs and risks (and withal, concomitant product risks) dramatically increase when the
development schedule is compressed. Thus, despite the basic management desirability to shorten
individual projects which compose a product development (and/or evolution), as suggested by Clancy
(1998), the utility of this technique has a lower bound when this increases risk and defects.
To summarize, an economic evaluation of investments particularly for items of questionable tangibility
should examine issues both quantitatively and qualitatively. It is not always simple to acquire the data
needed for a proper quantitative analysis; or at least, not a sufficient population of such data as to ensure an
analysis of reasonable value. In such a case, qualitative data can prove relevant and significant. This is
particularly poignant for complex products where fuzzy values are prevalent. Quantitative tools tend to be
more appropriate to economic valuation of traditional projects based upon hard tangible assets while,
qualitative (or a combination of quantitative and qualitative) tools tend to be more appropriately used for
soft or intangible assets. As such, our model uses a combination of quantitative and qualitative tools.
Thus, this combination of quantitative, qualitative and risk together with opportunities creates a much more
complex situation than that for which NPV was designed. The organization, after a complex product
implementation (many complex products consist of a system of systems), can arguably be a quite different
www.ejise.com
159
ISSN 1566-6379
organization than it was before. This degree of impact on the enterprise implies that the optimal model will
supply more than just a numeric measurement; but rather, it will show a strategic view of product value to the
enterprise, including qualitative values, information states, etc.
4. The model
We present a model for economic evaluation of complex products' desirability based on estimation of their
incremental contribution to corporate value. Standard financial models, such as NPV and ROI, are static,
have difficulty mapping between quantitative and qualitative data, are not oriented towards a view of systems
(or systems of systems) and not designed for constant updating via a functioning MIS. A model needed for
complex products must be dynamic. The large extent of risks and system uncertainty compels the
corporation to base decisions upon verifiable, event-related data, as supplied by both Financial and
Inventory Management Information Systems (see section 4.1.1, below), mapped with strategic Business
Goals and Product Plans. Notably, this complexity is layered on top of the Enterprise Projects Management
system. This basis, upon multiple management systems, allows application of the model to products and to
portfolios of products, at enterprise level.
Product planning generates a project or series of inter-related projects. In the framework of our model, at
product initiation, an initial estimation of economic profit from the product is performed based on projections
of costs, benefits, risks and schedules. All these projections are product-level; as a product management
process and not a part of the development (technical) process. This will usually be a part of the approval
process. Each year or period (whatever periodicity management chooses as optimal) a comparison needs to
be performed between projected and realized schedules, costs and benefits. Projections for the future are
then updated via new information from this period and new profit computations are executed. An intelligent
application of future projections adjustment demands cooperation between two parties; the firms financial
officers and software engineers. This collaboration is important for a thorough analysis from both
perspectives, finance and engineering, needed to complement each other.
During the evolutionary process of complex products there will be many versions and releases (Lehman &
Ramil, 2001). The computation as per our model, is intended to be realized from any point in time until any
point in time inclusive of any chosen timeframe.
The model is a corporate-level management instrument, for viewing a product (or portfolio). One of the
commonly used project management tools is Earned Value Management (EVM) (Boehm & Li, 2003; Ernst,
2006; DI-MGMT-81466, 2004; Lipke, 2002). EVM views tasks schedule, resources, requirements and costs.
Each and all together, are addressed as managerial aspects of the task that can be itemised. It is a bottomup project view and does not address overall product separating the project that creates the product or a
new version of it from an overall corporate view.
We now present the model; depicted by two flow graphs (Figure 1 and Figure 3). Figure 1 presents the basic
economic profit computation, with its information sources and processes. Figure 3 presents the iteration
process of updating profit computations, with relationship to business goals and overall product planning.
2
The graphical language used is standard notation, from Data Flow Diagramming techniques.
4.1 The basic economic profit computation

Figure 1 presents a flow graph, divided into three sections. The left section deals with costs acquisitions
(actual and projected), the middle section deals with acquisition and analysis of benefits (realized and
projected) and the right section deals with the economic profit computation, albeit with the updated and
dynamic inputs allowing needed model sophistication.
4.1.1 Costs acquisitions

Costs acquisition divides into two processes; quantitative cost elements and conversion of the costs into
monetary terms.
The first process recognizes quantitative cost elements and has two data sources. The first source is an online inventory system for software assets that shows enterprise software artefacts with their history and
2
A bubble is a process (always begins with a verb); a rectangle is a source or sink entity; a parallel with arced
ends represents a data stores and arrows represent flow (with heavy arrows depicting main flow path).
www.ejise.com
160
complexity, and the accumulated costs incurred (Ben-Menachem & Marliss, 2004). The second is a Products
Management MIS (see Figure 2). This system back-ends to MS-Projects for task tracking, providing data
relating to business goals, lifecycle events, present and projected resources, and business and product risks.
Note the absence of time as a controlling element. Time is provided for by any standard project management
tool (e.g., Primavera or MS-Projects). The Inventory system stores values concerning realized costs, while
the Products MIS shows projected costs.
The second process converts all costs (such as data stored in terms of hours, or other non-financial
terminology) into monetary terms for use in the profit computation. It is aided by a financial reporting system
(Ben-Menachem & Gavious, 2007).
4.1.2 Benefits acquisition and analysis

Costs are actual or projected values that are fairly straightforward to collect and simple to understand.
Benefits are much more complex, whether actualized already or projected (see section 3.2). The first
process is to capture system benefits. The main source for understanding benefits the definers of the system
intended is from the list of system goals. However, this will cover only those intended benefits defined as
primary goals of the system. Many systems have additional benefits that are indirect results of their
development. In many cases, these may even be difficult to discern. We recommend that the system be
examined, in a recursive relationship, to discover additional, indirect benefits.
These benefits are then critically examined and quantitative benefits are separated from qualitative.
Quantitative benefits are converted into monetary terms, via information from the financial reporting system
and supported by Expert opinion; while in parallel, the qualitative benefits are converted to metric values via
the Goals-Question-Metric Paradigm (Mashiko & Basili, 1997) or some equivalent tool. This analysis
provides additional value to the model as a management support for tracing system benefits development
and for auditing of systems value to the organization.
4.1.3 Computation of economic profit

The Economic Profit computation is executed based upon updated projected cash flows of costs and
benefits. Realized costs and benefits are relevant for the adjustment of projected costs and benefits,
however irrelevant for decision-making; i.e., they are no longer relevant for evaluating the viability of
continuing with the project. To account for risk and the time value of money, a risk-adjusted discount rate
needs to be intelligently estimated, with the required differences per industry or business environment, by the
firm's financial officers in collaboration with software engineers. This collaboration is needed for a risk
analysis from both perspectives of finance and engineering, resulting in a best estimation of a risk premium
to be accounted for in the cash flows' discount rate (Tockey, 2005).
www.ejise.com
161
ISSN 1566-6379
Costs
Benefits
The system
(recursively)
Products
Management
MIS
System
goals
Strategic
(direct) &
primary
indirect
Recognize
quantitative
cost
elements
Financial
experts
Differentiate
quantitative
from
qualitative
benefits
Recognize
system
benefits
Risk
adjusted
discount rate
Convert to
monetary
terms
Economic
profit
Convert
quantitative
to monetary
terms
Convert
qualitative
to metrics
Inventory
Financial
reporting
system
GQM
Parallel
analysis for
tracing &
audit
Expert
opinion
Figure 1: The basic economic profit computation

The processes described above need to be based upon sophisticated information processing tools. Until
recently, tools to provide corporate management with realistic, real-time data that maps both existing
systems and projects in progress were lacking. Recent development of a discipline called the Paradigm of
Change (henceforth PofC) (Ben-Menachem & Marliss, 2004) provides an integrated toolset designed for
data collection from projects, information management concerning projects (of all types, i.e., projects that
create new systems as well as those that update existing systems) and knowledge management for decision
support concerning complex products. Figure 2 shows the Products Management MIS focus of the PofC
(one of several possible foci that PofC provides).
www.ejise.com
162
Figure 2: Screen shot of products management MIS

Figure 1 includes a series of data-models (Products Management MIS and Inventory) drawn directly from
PofC and from financial reporting system which has information derived from PofC. In addition to these, the
Analysis, GQM and System functions base their operations on PofC-derived knowledge.
4.2 The iteration process

Figure 3 presents a superset of the economic profit computations presented in Figure 1. It shows the
evolutionary profit computations process, automatically updated with aid of the information system; hence
making it a useful enterprise management tool.
The Product Plan and the Business Goals from which the product plan is derived, are general inputs to this
process. The product plan has a defined scope and objective. Scope is critical to comprehend economic
profit computation, vis--vis the goals defined by the business. Project plans are created by engineering staff
(engineering management) in terms of performance, time (milestones), risks management and budget. The
Projected Costs and Projected Benefits in Figure 3 are projected at product initiation, but in realistic
practice today, seldom revisited categorically for continual feedback as to product viability.
The basic, initial economic profit (I) computation is performed at initiating project (the project that initiates
the product) start, usually as part of the process of product approval. Each period (as chosen by
management) a new computation is performed based on adjusted Cash Flow (CF) projections (economic
profit (Sn)). As shown, the process of performing these computations is now much simplified by the fact that
the input data for the computations are all based upon the PofC Management Information System. The
succeeding computations are performed iteratively per period. Note the two feedback loops in Figure 3. The
inner loop (dashed line) is internal to the model process. It shows that the computed profit is based upon
constantly updated data/information from the MIS and that each new computation is, itself, also stored in the
MIS for future use. Thus, the accumulation of computed profits becomes a knowledge base with which upper
management can deal for constant updating of business goals processing.
www.ejise.com
163
ISSN 1566-6379
Notably, this becomes a positive feedback loop (as opposed to passive feedback) because it represents
Past-Present-Future value of model use. Accounting systems, for instance, show a picture of the present
based upon accumulated data from the past. However, such normative management information is
augmented by showing the combined processes of profit evaluations at every action-point during product
evolution. Additionally, these action-points can be compared one with others, across projects and across
products, so that decision-points and action-points can be aggressively compared for enhancing the
processes of making improved business decisions. This leads to Future, where the Past and the Present
combine positively to enhance both the business model itself and the processes of improving the modelling
technique.
Finally, as shown in Figure 3, the model contains an element called defects. Every technology-rich project
creates defects while in production. This is the state-of-the-art and the inherent nature of all technology-rich
projects. Their quantity can be limited by very careful development processes, and their effects can be
minimized if technology management so desires. These are choices with associated short-term costs;
however it is generally accepted that the long-term corporate benefits justify them. Product defects may
interfere with an ability to satisfy critical requirements or key functionality and always negatively affect
customer satisfaction. Defects also negatively affect schedules and future product cash flows. All this
negativity must be reflected in projections.
Some project management methods can deal with estimations of defects and resources consumed, such as
parametric estimating, backfiring or several function point estimation techniques. This relates to specifics of
software coding and is outside the scope of this article.
Projected
costs
Projected
benefits
Products
Management
MIS
The
system
Defects
Product
plan
CF
projections
Economic
profit (I)
Realized
CF (Sn)
Business
goals
Adjusted
projections
Economic
profit (Sn)
Defects
iterations
Knowledge-base of economic
profit processes
Figure 3: The iteration process

The suggested model helps learn from history (mistakes) in estimating product economic desirability, where
costs are typically underestimated and benefits are typically overestimated (Tockey, 2005). Experience
improves firm's costs and benefits analysis and hence its investment decisions. Beyond managements need
to perceive which resource investments may produce an optimal result is now added a regulatory
requirement for lifecycle traceability. Sarbanes-Oxley Act of 2002 (Section 203) requires that IT be made
transparent to the business and fit within mainstream business management. Specific assertions have a
general relationship to assertions about internal controls for both manual and information processing
systems. An assertion can either be explicit or implicit and based upon specific procedures that can be
tested and used to evaluate the assertions made by management. Furthermore, Assertions made
regarding disclosure/internal controls can be evaluated by testing stated or measurable criteria. The criteria
are based upon specific assurance objectives such as business goals that are traced via change
management, mapped from management assertions to products and product components.
www.ejise.com
164
To accomplish these regulatory requirements, IT and business goals need to align and this alignment must
be traceable, throughout the entire product lifecycle that is, from the first requirements statement until the
product is finally retired from all use. Use of automated requirements gathering and analysis tools, together
with design and testing traceability, allows auditability of this alignment, both immediately and long-term.
Economic evaluation, performed periodically via auditable information, from a traceable resource (MIS) and
recorded for continual re-evaluation, endows corporate management with verifiable tools to affirm this
regulatory requirement. Figure 1 shows this feedback and how this 'loop' is utilised for weaving the regulatory
requirement into the corporate fabric requirements, audits, economic evaluation, new product planning,
etc.
5. Conclusion
Products and product portfolios should be economically valued, using economic terminology and standard
financial computations. Economic models, common to the financial-world, apply to information systems
(software-rich products); for they can be made to apply with adjustments. The application of economic
models inherently enhances a corporation's long-term viewing capabilities. Software-rich products need to be
managed with a view extending significantly farther than competitors conceived. Notably, many analysts
have claimed that the primary driving force to market power is a long-term view.
Organizations need a long-term knowledge base, to attain a long-term view. The on-going management
process, with all its decision-points, increases the necessity for long-term product information. Moreover,
todays regulatory environment (e.g., Sarbanes-Oxley Act of 2002) necessitates the availability of such
information.
Withal, how can this long-term view be consistently achieved, even in corporate environments that have not
been previously built upon such concepts? Or, what can a corporation do, when it perceives that this kind of
product view is not yet its corporate strength, and wishes to enhance capabilities in this area?
This article discusses these issues and provides a dynamic model for economic evaluation of complex
products desirability, via an inter-disciplinary approach, by mapping products, with the needs and exigencies
of corporate management, product managers and software engineers. The model facilitates real time
reporting via an information system designed for management of Products, Portfolios and Projects. This
dynamism is shown to be management crucial because of the extreme uncertainty embedded in projections
of costs, benefits, risks and timeframes associated with software-rich, complex products. As an additional
significant benefit, the model fulfils Sarbanes-Oxley Act of 2002 requirements for management assertion
traceability of valid and accurate measures, with auditability.
The model is designed to be based upon quality information, from automated data-gathering utilities, that is,
the information is both of high quality and highly auditable, allowing corporate management to leverage this
information to maximum benefit. What is no less significant is that this information can be used consistently,
over related products, and to make long-term comparisons between product lines, while taking all their
aspects properly into account.
These aspects co-joined, from Sarbanes-Oxley, back through multiple products, over myriad versions, and
through automated requirements, design and testing tools, all combine to form an auditable management
feedback loop that can be leveraged at multiple corporate management levels. The paper represents a
significant step towards quality product decision-making via a model that is meaningful, while also useful as
it is leveraged through an automated tool set.
References
Armour, P. G. (2005) "Project Portfolios: Organizational Management of Risk", Communications of the ACM, Vol 48, pp.
17-20.
Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (ASD C3I). (2002). Repairing
Latent Year 2000 Defects Caused by Date Windowing. Investment and Acquisition Directorate, US Department of
Defense, Version 1.0, April.
Asuaga, A. (2001) "Engineering a New Society", IEEE Computing, June, pp. 102-104.
Ben-Menachem M., & Gavious, I. (2007) "Accounting Software Assets: A Valuation Model for Software", Journal of
Information Systems of the American Accounting Association, Vol 21, No.2, pp. 117-132.
Ben-Menachem, M., & Marliss, G. (1997) Software Quality: Producing Practical, Consistent Software, Intl Thomson
Computer Press.
www.ejise.com
165
ISSN 1566-6379
Ben-Menachem, M., & Marliss, G. (2004) "Inventorying Information Technology Systems: Supporting the "Paradigm of
Change"", IEEE Software, Vol 21, pp. 34-43.
Boehm, B. (1981) Software Engineering Economics, Englewood Cliffs, NJ: Prentice Hall PTR.
Boehm, B. (1997) COCOMO II Model Definition Manual. University of Southern California, Version 1.4.
Boehm, B. & Li, G. H. (2003) "Value-Based Software Engineering: A Case Study", Computer, Vol 36, No.3, pp. 33-41.
BPM Forum. (2004) Software Drain or Business Gain: Assessing Application Value, Relevance and Cost to Your
Company. http://www.bpmforum.org/PDF/Software_Drain.pdf; December.
Broekman, B., & Notenboom, E. (2003) Testing Embedded Software, Addison Wesley.
Brooks, F. (1995) The Mythical Man-Month, Addison Wesley.
Clancy, T. (1998) CHAOS, Standish Group.
DI-MGMT-81466. (2004) Contract Performance Report. Department of Defense, December 2004.
Ernst, K. D. (2006) Earned Value Implementation Guide. Department of Defense, October 2006.
Greiner, L. (2003) "Analytics Unlock the Secrets to ROI", Computing Canada, Vol 29, pp. 16-17.
Heisenberg, W. (1991) The Copenhagen Interpretation of Quantum Theory; as reproduced in The Treasury of Physics,
Astronomy and Mathematics, ed. Timothy Ferris, Little, Brown and Company, pp. 86-96.
Lehman, M. M. (1990) "Uncertainty in Computer Applications", Communications of the ACM, Technical Correspondence,
V33N5, pp. 584-586.
Lehman, M. M., & Ramil, J. F. (2001) "Rules and Tools for Software Evolution Planning and Management", Annals of
Software Engineering, Vol 11.
Lipke, W. H. (2002) EVM and Software Project Management - Our Story. Cross Talk.
Longstaff, T. A., Chittister, C., Pethia, R., & Haimes, Y. Y. (2000) "Are we forgetting the risks of information technology?",
Computer, December, pp. 43-51.
Mashiko, Y., & Basili, V. (1997) "Using the GQM Paradigm to Investigate Influential Factors for Software Process
Improvement", Journal of Systems Software, Vol 36, pp. 17-32.
Sarbanes-Oxley Act of 2002. Available at:
http://fl1.findlaw.com/news.findlaw.com/hdocs/docs/gwbush/sarbanesoxley072302.pdf
Serich, S. (2005) "Prototype Stopping Rules in Software Development Projects", IEEE Transactions on Engineering
Management, Vol 52, No.4, pp. 478-485.
Taylor, A. (2001) IT Projects Sink or Swim. British Computer Society Review, September.
Tockey, S. (2005) Return on Software, Addison Wesley.
Turnbull, I. (2003) "Finding the Return on HR Technology Investments", Canadian HR Reporter, Vol 16, pp. 17-18.
Verhoef, C. (2002) "Quantitative IT Portfolio Management", Science of Computer Programming, Vol 45, pp. 1-96.
www.ejise.com
166

EJISE Volume 11 Issue 3

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EJISE Volume 11 Issue 3

Uploaded by

Copyright:

Available Formats

B2C e-Commerce Success: a Test and Validation of a Revised

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

Table 1: Dimensions of IS Success

3.1 Continuance intention

3.2 User satisfaction

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

3.3 Perceived usefulness

3.4 Loyalty incentives

3.6 System quality

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

3.7 Information quality

3.8 Service quality

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

Figure 1: B2C E-commerce success model

4.1 Instrument design

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

4.2 Sample design

4.3 Pilot study

4.4 Data collection and analysis procedure

5. Data analysis and results

5.1 Demographic profile

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

5.2 Internet and e-commerce experience

5.3 Instrument validity, reliability and refinement

5.3.1 Construct validity

Use a minimum eigenvalue of 1 as a cutoff value for extraction;

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

Table 4: Refined Instruments

Original Instrument (No. of Items)

Refined Instrument (No. of Items)

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

5.3.2 Instrument reliability

User Satisfaction (US)

5.4 Descriptive statistics for B2C e-Commerce success dimensions

5.5 Hypothesis testing

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

System Quality (SQ)

System Quality (SQ)

System Quality (SQ)

System Quality (SQ)

Service Quality (SS)

Service Quality (SS)

Service Quality (SS)

Service Quality (SS)

The supported hypotheses are illustrated in Figure 2 below.

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

Figure 2: Refined e-Commerce success model

6. Discussion and implications

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

7. Limitations and further research

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)

Academic Conferences Ltd

Irwin Brown and Ruwanga Jayakody

Appendix 1: Instrument Measures

Perceived Usefulness (PU)

Information (Content) Quality (IQ)

Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008 (167-184)