MACROERGONOMIC ORGANIZATIONAL QUESTIONNAIRE

SURVEY (MOQS)

Pascale Carayon and Peter Hoonakker

University of Wisconsin-Madison

Background and Application

Questionnaire surveys can be used to collect information on a range of

ergonomics variables (Salvendy and Carayon 1997). Macroergonomic organizational
questionnaire surveys collect information on various aspects of the work system
(Carayon and Smith, 2000): Tasks, organizational conditions, environmental issues, tools,
technologies, and individual characteristics. In addition, macroergonomic organizational
questionnaire surveys collect information on various outcomes, such as quality of
working life (e.g., job satisfaction), physical and psychological stress, physical and
mental health, performance and attitudes (e.g., intention to turnover).
Macroergonomic organizational questionnaire surveys can be a useful tool at
several stages: Diagnosis stage, benchmarking an organization on key characteristics of
interest, evaluating the impact of a change on key characteristics, monitoring worker
opinions during the implementation of a change.
Carayon and Hoonakker (2001) have emphasized the following issues for
macroergonomic organizational surveys: Objectivity/subjectivity, reliability and validity,
development of the questionnaire, and conducting the survey. At the questionnaire
development stage, it is important to clearly define the concepts to be measured and to
explore the range of questions that can be used to measure those concepts. In particular,
attention should be paid to the measures degree of objectivity/subjectivity, i.e. the degree
to which cognitive and emotional processing influences answers to the questions
(Carayon and Hoonakker 2001).

Procedure

Particular attention should be paid to the development of questionnaires. The

methods used to develop, implement and use questionnaires are very important for the
quality and usefulness of the data collected. Before developing a questionnaire survey, it
is important to clearly specify the objective of the questionnaire: What will the
questionnaire be used for?
Carayon and Hoonakker (2001) have defined five stages for developing a
questionnaire survey:
Step 1 - Conceptualization
Step 2 - Operationalization
Step 3 - Sources of questionnaire
Step 4 Constructing the questionnaire
Step 5 - Pre-testing of the questionnaire.

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 1

Each step is described in detail in Carayon and Hoonakker (2001). Table 1 describes the
major questions to ask for each step and lists background information or materials that
can be useful to complete successfully each step.

The actual conduct of a macroergonomic organizational questionnaire survey

involves the following steps (Carayon and Hoonakker 2001; Church and Waclawski
2001):
Step 1 Pooling resources
Step 2 Communicating objectives
Step 3 Administering
Step 4 Analyzing and interpreting
Step 5 Delivering results
Step 6 Transferring and action planning
Each step is described in detail in Carayon and Hoonakker (2001) and in Church and
Waclawski (2001). Table 2 describes the major questions to ask for each step.

The process for developing and conducting a macroergonomic organizational

questionnaire survey can be summarized by the following questions:
- WHAT? What are the objectives? What measures will be used? How reliable and
valid are the measures?
- HOW? What is the process used to develop the questionnaire? What is the
process used to conduct the survey? How is feedback provided to the organization
and its employees?
- WHO? Who is to participate in the survey? Who is involved in developing and
conducting the survey? Who should be informed at what step?
- WHEN? When will the survey start and end? What is the timeline for the entire
process?
- WHERE? Where will the survey be conducted? Where will the data be stored?

Advantages

Surveys allow the researcher to obtain large amounts of data from large numbers
of people at relatively low cost and relatively fast (Sinclair 1995). In addition,
macroergonomic organizational questionnaire surveys provide structured data that can be
easily scored, analyzed and compared.

Disadvantages

A major issue in the development of a macroergonomic organizational

questionnaire survey is to identify the objectives of the survey and, therefore, define the
concepts to be measured in the survey. It is sometimes difficult to know how to ask the
right questions and the right response categories. This is the reason why pre-testing a
questionnaire is important. Two major weaknesses of questionnaire surveys are the
limited space to ask questions, and the limited time for questions.

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 2

Examples

A couple of examples are shown to demonstrate the potential uses of the

macroergonomic organizational questionnaire survey.

First example: Evaluating employee opinion regarding a technology implementation

(Carayon and Smith 2001)
Figure 1 shows results of an employee questionnaire survey performed before the
implementation of an information technology. The results of the survey showed many
negative opinions regarding the technology implementation: lack of information received
about the technology, lack of inputs regarding the design and implementation of the
technology, lack of understanding of the technology impact on ones job, and negative
attitudes toward the technology. This information provides diagnosis data that
management can use in order to improve the technology implementation. For instance,
based on these data, management might consider various means for communicating with
employees regarding the design and implementation of the technology.

Technology Issues

100
90
80
70
60 54
51
48
50
40 33 32
30
20
10
0
Information Received Inputs Regarding Inputs Regarding System Effect on Job Attitude Toward
about System Design Implementation System

Figure 1 Employee opinions regarding a technology implementation (measured before

the implementation)

Second example: Impact of a work organization intervention (Carayon, Haims et al.

2000)

A series of work organization interventions aimed at improving quality of

working life were implemented in three groups of computer users and office workers in
an organization. Group 1 received the intervention right after R1-Round 1, whereas

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 3

Groups 2 and 3 received the intervention after R2-Round 2. Data was collected at three
times separated by about 6-8 months (R1-Round 1, R2-Round 2, R3-Round 3). Several
elements of the work system and outcomes were evaluated in this study. Figure 2 shows
data for one element of the work system, i.e. pace control. Figure 2 shows that group 1
employees perceived an increase in pace control after the work organization intervention,
whereas pace control did not increase for group 2 employees. Group 3 employees
actually reported decreased pace control. When presenting the results to management and
employees, the increase in pace control for group 1 was confirmed to be a result of the
work organization intervention. And the decrease of pace control in group 3 was
explained by major organizational changes occurring in this group at the time of the
Round 3 data collection.

Pace control over time

7.0

6.0

5.0
Pace control

4.0

Group
3.0

Group 1
2.0
Group 2

1.0 Group 3
R1 R2 R3

Figure 2 Employee perceptions of pace control over time

Reliability and Validity

The reliability and validity of macroergonomic organizational questionnaire surveys have
been studied extensively and found satisfactory in many studies. Specific data on
reliability and validity can be found for various forms of MOQS (see, for example, Cook
et al., 1981).

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 4

In the study used as the second example above (Impact of a work organization
intervention by Carayon et al., 2000), the various measures of psychosocial work factors
were found to have reliability Cronbach-alpha scores comprised between .45 and .94. A
Cronbach-alpha score of .70 is accepted as an appropriate minimal level (Nunnaly, 1978).
In the Carayon et al. (2000) study, 20% of the psychosocial work factors measures had
Cronbach-alpha scores below .70. The main reason for the low reliability scores for these
measures was the low number of questionnaire items comprising the measures.
Cronbach-alpha scores are dependent on the number of questionnaire items comprising a
measure, and many of these measures with low reliability scores were comprised of only
2 items. In this study, many of the measures of psychosocial were found to change over
time because of the work organization interventions (see Figure 2 for data on the
psychosocial work factor of pace control). This result is an indication of the validity of
these measures of psychosocial work factors. A longitudinal study of psychosocial work
factors, ergonomic factors and health by Hoonakker et al. (1998) showed the predictive
validity of these variables on absenteeism among a large sample of Dutch construction
workers. For instance, positive experienced health was related to short absenteeism and
lots of turbulence on the job were related to more frequent absences.

Because much effort, expertise and time are necessary to ensure the quality of
questionnaire surveys, it is recommended to use established questionnaires as much as
possible (see, for example, Cook et al. for various questionnaires relevant to
macroergonomics). The following questionnaire surveys have been shown reliable and
valid, and have been used extensively in various macroergonomic studies:
- NIOSH Job Stress Questionnaire
- University of Wisconsin Office Worker Survey
- Job Content Questionnaire by Karasek.

Tools Needed

Macroergonomic organizational questionnaire surveys can be conducted using

different methods. With the increasing use of Internet and other electronic
communication technologies, more and more questionnaire surveys are done via
electronic methods. One can distinguish between several electronic methods:
survey embedded in an email
survey attached as a document
survey attached as a program (EXE. File)
web-based surveys The advantages of electronic surveys as compared to mail
surveys include the following:
easy access to (worldwide) samples
speed and response time
costs (time and money, only constructing and posting)
reducing errors (e.g. data-entry)
high response quality
more completed questionnaires
higher probability to answer open ended questions

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 5

 flexibility (respondents can choose how and when to respond)
ease of administration
ease of formatting (color, sound, images)
skip patterns, pop-up boxes etc

The disadvantages of electronic surveys as compared to mail surveys include the

following:
coverage error/bias (limited and biased population, class, race, age, income &
gender)
sampling error
measurement error (PC (il)literacy, multiple copies)
non response error
anonymity
non-deliverability (20%, Bachman et al., 1996, 1999; 19% and 24% Weible and
Wallace, 1998)
security.
Table 3 lists studies that have used mail surveys and various forms of electronic surveys.
The results of these studies show that (1) mail surveys tend to lead to higher response
rates than email and/or web surveys, and (2) response rates to email have gone down over
time. However, web-based are getting better results (Cobanoglu et al., 2001). Therefore,
when deciding whether to conduct a mail survey or an electronic survey, attention should
be paid to the various factors that contribute to response rates: Salience of the issue
addressed by the survey, pre-notification, personalized communication, length of survey,
sponsorship and management commitment, and follow-up (Dillman 2000).

References

Babbie, E. (1990). Survey research methods. Belmont, CA: Wadsworth.

Bachmann, D., Elfrink, J. and Vazzana, G. (1996). "Tracking the progress of e-mail vs.
snail-mail." Marketing Research 8 (2), 30-36.
Bachmann, D., Elfrink, J. and Vazzana, G. (1999). "E-mail versus snail-mail face off in
rematch." Marketing Research 11 (4), 10-15.
Carayon, P., Haims, M.C. Hoonakker, P.L.T. and Swanson, N.G. (2000). Intervention
research for reducing musculoskeletal injuries. Proceedings of the IEA
2000/HFES 2000 Congress, Volume 2 (169-171). Santa Monica, CA: Human
Factors and Ergonomics Society
Carayon, P., & Smith, M. J. (2000). Work organization and ergonomics. Applied
Ergonomics 31, 649-662.
Carayon, P. and P. Hoonakker (2001). Survey design. In W. Karwowski (Ed.),
International encyclopedia of ergonomics and human factors (pp. 1899-1902)
London: Taylor & Francis.
Carayon, P., & Smith, P. D. (2001). Evaluating the human and organizational aspects of
information technology implementation in a small clinic. In M. J. Smith, & G.
Salvendy (Eds.), Systems, social and internationalization design aspects of
human-computer interaction (903-907). Mahwah, NJ: Lawrence Erlbaum .

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 6

Carmines, E. G., & Zeller, R. A. (1990). Reliability and validity assessment. Beverly
Hills ,CA, Sage.
Church, A. H. & Waclawski, J. (2001). Designing and Using Organizational Surveys.
San Francisco: Jossey-Bass.
Cobanoglu, C., Warde, B. and Moreo, P.J. (2001). A comparison of mail, fax and web-
based survey methods. The Market Research Society 43 (4), 441-52.
Converse, J. M., & Presser, S. (1986). Survey Questions: Handcrafting the Standardized
Questionnaire. Beverly Hills, CA, Sage.
Cook, J. D., Hepworth, S.J., Wall, T.D. and Warr, P.B. (1981). The Experience of Work.
London: Academic Press.
Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. New
York: Wiley.
Hoonakker, P.L.T., Dierendonck, D. van, Molen, H.F. van der, Ginkel, A. van (1998).
The relation between experienced workload, working conditions, health and
absenteeism in construction industry, " in Human Factors in Organizational
Design and Management VI, edited by P. Vink, Elsevier Science Publishers, The
Netherlands, pp.683-688.
McDowell, I., & Newell, C. (1987). Measuring health: A guide to rating scales and
questionnaires. Oxford, UK: Oxford University Press.
Nunnaly, J. C. (1978). Psychometric theory. New York: McGraw Hill.
Salvendy, G., & Carayon, P. (1997). Data-collection and evaluation of outcome
measures. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp.
1451-1470). New York: Wiley.
Sinclair, M. A. (1995). Subjective assessment. In J. R. Wilson, & E. N. Corlett (Eds.),
Evaluation of human work - A practical ergonomics methodology (pp 68-100).
London: Taylor & Francis.
Weible, R., & Wallace, J. (1998). The Impact of the Internet on data collection.
Marketing Research 10 (3): 19-27.

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 7

Table 1 Development of a macroergonomic organizational questionnaire survey
STEPS
Step 1 Conceptualization
Step 2 Operationalization
Step 3 - Sources of
questionnaire
Step 4 Constructing the
questionnaire
Step 5 - Pre-testing of the
questionnaire
To be published in the Handbook of Human Factors and Ergonomics Methods - 2002
QUESTIONS TO ASK BACKGROUND
What concepts will be measured by the Babbie (1990)
macroergonomic organizational questionnaire survey?
What elements of the work system will be
evaluated: tasks, organizational conditions,
physical environment, tools and technologies,
individual characteristics?
What outcomes will be evaluated: quality of
working life, physical and psychological stress,
physical and mental health, performance and
attitudes?
What are the objectives of the survey and how do the
concepts to be measured fit with the objectives?
What are the dimensions of each concept? Babbie (1990)
Check for overlap.
Review previous work.
What questionnaire surveys are available that can be Cook et al. (1981)
used? Examples of questionnaire surveys:
Office Worker Survey University of
Wisconsin-Madison
NIOSH Job Stress Questionnaire
Karaseks Job Strain questionnaire
What forms of questions will be used? Converse and Presser (1986)
What are the rating scales? Dillman (2000)
How should be questionnaire be organized (i.e. order of
questions, instructions, layout)?
Who will participate in the pre-test?
What are the objectives of the pre-test (e.g., check for
clarity of questions, test the questionnaire format,
assess the duration of the questionnaire)?
8
Table 2 Conducting a macroergonomic organizational questionnaire survey
STEPS
Step 1 Pooling resources
Step 2 Communicating
objectives
Step 3 Administering
Step 4 Analyzing and
interpreting
Step 5 Delivering results
Step 6 Transferring and
action planning
To be published in the Handbook of Human Factors and Ergonomics Methods - 2002
QUESTIONS TO ASK
Are the organizations management and key stakeholders committed to the successful
completion of the questionnaire survey?
Who are the key stakeholders?
How should be they be involved and informed?
What will be communicated?
How will the communication occur? How often? To whom?
How will the survey be presented to the respondents? What are the expected benefits and
costs of the survey for the respondents?
What is the best time to conduct the survey?
What procedures will be used to administer the survey?
What methods will be used to collect the data (e.g., paper versus electronic survey)
What softwares will be used to enter the data, to analyze the data, and to present the data
analysis?
What statistical methods will be used? How do the statistical methods help achieve the
survey objectives?
To whom will the results be presented? When? In what order?
How will the report be structured? Who will read the report?
How will confidentiality of respondents be assured?
How will the integrity of the results be kept?
How is commitment to actions secured?
What follow-up activities are necessary to insure that the data is used for planning and
implementing actions?
Is any follow-up survey scheduled? If yes, how will this be done? When?
9
Table 3 Studies using mail surveys and various forms of electronic surveys [For a complete list of references of these studies, please
contact the authors)

Author(s) Sample Survey Method Sample Response Response Response quality

Topic Size Rate Time
(days)
Kiesler & Sproull (1986) Employees of a Fortune Corporate Mail 115 67% 10.8 E-mail had fewer mistakes and a
500 company communication Email1 115 75% 9.6 higher completion rate
Parker (1992) Employees of AT&T Internal Mail 70 38% NA NA
communication Email 70 68% NA NA
Schuldt & Totten (1994) Marketing and MIS professors Shareware Mail 200 56.5% NA NA
copying Email 218 19.3% NA NA
Mehta & Sivadas (1995) Usenet users Internet Mail 309 56.5% NA Both groups had similar number
communication Email 182 54.3% NA of item omissions, but e-mail
respondents wrote more
Tse et al (1995) University population Business ethics Mail 200 27% 9.8 No significant difference in
Email 200 6% 8.1 number of item omissions
Bachman et al Business school deans TQM Mail 224 65.6% 11.2 E-mail respondents were more
(1996) Email 224 52.5% 4.7 willing to answer open-ended
questions
Weible & Wallace MIS professors Internet use Mail 200 35.7% 12.9 NA
(1998) Fax 200 20.9% 8.8
Email 200 29.8% 6.1
Web form 200 32.7% 7.4
Schaefer & Dillman University faculty Unknown Mail 226 57.5% 14.4 E-mail surveys had fewer item
(1998) Email 226 58.0% 9.2 omissions and longer answers to
open-ended questions
Bachman et al Business school deans and TQM Mail 250 66.0% 18.3 No differences in response
(1999) division chair persons Email 250 19.1% 4.3 patterns. E-mail respondents
were more willing to answer
open-ended questions.
Sheehan & McMillan, Creators of health related web Values of site creators, Email 2 834 47% 5.0 More salient issues and
(1999) sites site purpose and funding (individual) prenotification did increase
response rates. Reminder
increased response rates, varying
Faculty, staff and students Attitudes toward on-line Email (batch) 580 47% 4.6 from 23-48%
privacy
Individuals with personal e-mail Attitudes and behaviors Email (merge) 3,724 24% 3.6
accounts associated with on-line
privacy
Dommeyer & Moriarty, CSUN students Attitudes towards binge Email (embedded) 150 37% 4.3 No significant difference in
(2000) drinking Email (attached) 150 8% 5.7 number of item omissions.
Couper, M.P., Blair, J. & Employees in several government Organization climate Agency A: mail 2,699 68% NA E-mail response rate was much
Triplett, T. (2000?) statistical agencies Agency A: email 2,699 37%

1
Mail = postal mail; E-mail, unless defined otherwise = questionnaire embedded in the E-mail
2
Respondent were given the option to return a paper copy. 3% made use of this option.

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 10

 Agency B: mail 790 76% NA lower, mostly due to technical
Agency B: email 396 63% problems (different e-mail
Agency C: mail 266 74% NA software). No significant
Agency C: email 265 60% difference in number of item
Agency D: mail 216 75% NA omissions
Agency D: email 221 53%
Agency E: mail 216 76% NA
Agency E: email 215 55%
Cobanoglu et al Hospitality professors Hospitality Mail 100 26% 16.5 80.7% (completed surveys)
(2001) education Fax 100 17% 4.0 76.4% (completed surveys)
Email (WBS) 100 44% 6.0 81.4% (completed surveys)

To be published in the Handbook of Human Factors and Ergonomics Methods - 2002 11