Instrument Validation: Data Types, Scales, & Instruments

Detmar Straub Georgia State University

Graphics available at: detmar straub.com

A short course in ensuring that measurement error is within acceptable scientific bounds
1

Agenda
1.

2. 3. 4. 5. 6.

7.

a. b. c.

What are the different types of data and why it matters Scales Instruments and CMV Instrument validity versus other critical validities Formative versus reflective measures Validity roadmap
Content validity Reliability Construct validity

Why we should care about having valid instruments

Presentation is available for downloading at: detmarstraub.com

2

1.

What are the different types of data and why it matters

Which came first? The Chicken or the Egg (Design of a study or choice of data type/statistical test)

1.

What are the different types of data and why it matters

2.

Scales

[Examples are Likert (agree scales like #1-#4) , fully anchored scales (#5), & Semantic Differentials (end points anchored by phrases at bottom of page, #1-#2) ]

Demonstration via a security questionnaire used in previous studies.

3.

Instruments and CMV (Common Methods

Variance or Bias)

The Technology Acceptance Model

I love 'em!

I hate 'em!

The Technology Acceptance Model

PU = Perceived Usefulness (of IT) PEU = Perceived Ease of Use (of IT)

+
PU

Attitudes

Intentions

USE

+
PEOU

+
-adapted from Davis' Technology Acceptance Model (Davis, 1986, 1989)

3.
Constructs
Intention to Use System
1 2 3 4 5 6 7

Instruments and CMV (Common Methods

Variance or Bias)
Non-Randomized Presentation of Items
I will very likely use CHART-MASTER. I will probably use CHART-MASTER. I intend to use CHART-MASTER. I expect my company to use CHART-MASTER frequently. Using CHART-MASTER in my job would enable me to accomplish tasks more quickly. Using CHART-MASTER would improve my job performance. Using CHART-MASTER in my job would increase my productivity. Using CHART-MASTER would enhance my effectiveness on the job. Using CHART-MASTER would make it easier to do my job. I would find CHART-MASTER useful in my job. I am very likely to try out CHART-MASTER.

Randomized Presentation of Items

2 4 5 8 9 11 15 18 19 21 23 31 34

I will very likely use CHART-MASTER. Using CHART-MASTER in my job would enable me to accomplish tasks more quickly. I expect my company to use CHART-MASTER frequently. I would find it easy to get CHART-MASTER to do what I want it to do. Using CHART-MASTER would make it easier to do my job. I would find CHART-MASTER easy to use. I am very likely to try out CHART-MASTER. Using CHART-MASTER would improve my job performance. It would be easy for me to become skillful at using CHART-MASTER. Using CHART-MASTER in my job would increase my productivity. Learning to operate CHART-MASTER would be easy for me. My interaction with CHART-MASTER would be clear and understandable. I would find CHART-MASTER useful in my job. I will probably use CHART-MASTER. I would find CHART-MASTER to be flexible to interact with. I intend to use CHART-MASTER. Using CHART-MASTER would enhance my effectiveness on the job.

Perceived Usefulness

8 9 10 11

12

Perceived Ease of Use

13 14 15 16 17

Learning to operate CHART-MASTER would be easy for me. I would find it easy to get CHART-MASTER to do what I want it to do. My interaction with CHART-MASTER would be clear and understandable. I would find CHART-MASTER to be flexible to interact with. It would be easy for me to become skillful at using CHART-MASTER. I would find CHART-MASTER easy to use.

35 40 41 46

Table 2. Item Ordering Threats to Construct Validity through Common Methods Bias

4.

Instrument validity versus the other critical validities

Based on: Straub, D., Boudreau, M.-C., and

Gefen, D. 2004. "Validation Guidelines for IS Positivist Research," Communications of the AIS (13:24), 380-427.

GIGO (garbage in, garbage out)

Instrumentation Validity Internal Validity Statistical Conclusion Validity
Mathematical relationships between the presumed constructs are assured within certain degrees of confidence; rival hypotheses are possible; constructs may not be real and reliable; instrument may be measuring the wrong content. Rival hypotheses are ruled out; constructs may not be real and reliable; instrument may be measuring the wrong content. Mathematical relationships between the presumed constructs are assured within certain degrees of confidence; rival hypotheses are ruled out; constructs may not be real and reliable; instrument may be measuring the wrong content. Mathematical relationships between the constructs are assured within certain degrees of confidence; rival hypotheses are ruled out; constructs are likely real and reliable; instrument is likely measuring the right content.

Constructs are likely real and reliable; instrument is likely measuring the right content.

Rival hypotheses are ruled out; constructs are likely real and reliable; instrument is likely measuring the right content.

Time
Figure 1. Validity Touchstones (Altered Somewhat from Straub, 1989) (Legend: Green is preferred path; yellow is cautionary; red is least desirable path)

10

5.

Formative versus reflective measures

Source: Petter, S., Straub, D., and Rai, A. 2007. "Specifying Formative Constructs in IS Research," MIS Quarterly (31:4, December), 623-656.

11

A gingerbread cookie man analogy

Not Interchangeable

Interchangeable (with estimated error)

Servqual: Usually Specified as Reflective..Is It?

Item No. 1 2 3 4 5 6 7 8 9 10 11 12 Item Description Convenient operation hours Perform service right Has user interest in heart Willingness to help Capability to do what is promised Knowledge to do job well Sincere interest toward problem solving Responsiveness to user Capability to keep user information safe Consistent courteousness Capability to give user individual attention Capability to understand user needs Factor 1 0.807 0.835 0.865 0.860 0.849 0.848 0.863 0.881 0.749 0.870 0.818 0.790

If, in their factor analysis, the researchers had forced the SPSS software to find more factors, the construct will start to break apart.

One Construct (Omission of Security Actions): Measured Reflectively & Measured Formatively
Reflective Measures of Omission of Security Actions
Code
OMIT7 OMIT13 OMIT57

Item
I take measures to protect my information from security violations: Neveralways I take precautions against information security violations: Never..always I use information security protections: Never..always

Loading
0.9383 0.9514 0.9357

T-stat
53.2516 70.2334 50.1780

Composite reliability .959

Scale developed according to Ajzens (2002) recommendations for measuring intentions.

Formative Measures of Omission of Security Actions

Code Item Weight 0.6051 0.5162 -0.1002 T-stat 18.2871 15.8049 4.7784 PWDOMIT Computer logs of password changes UPOMIT BACKOMI T Computer logs of security updates Computer logs of backups

6.

A validity roadmap

15

Roadmap

16

Roadmap

17

6a. Content validity

Universe of all possible measures for a given construct

Drawing representative measures for an instrument

Figure 2. Pictorial Model of Content Validity

18

6a. Content validity

Content validity is about the elements or dimensions that you are capturing in your scale items and whether you have captured the essence of the construct or whether you have left something out. Generally assessed through expert panels, but Lawshe (1975) offers a quantitative approach [Lawshe, C.H. "A Quantitative Approach to
Content Validity," Personnel Psychology (28) 1975, pp 563-575.]

The essential content validity question is: Is Lawsches CVR (content validity ratio) statistically significant at some alpha protection level like .05? Significance is interpreted to mean that more than 50% of the panelists rate the item as either essential or important.

19

6b. Reliability

Latent Reflective Construct A

?
?

Latent Formative Construct B

Latent Reflective Construct C

Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 6

Measure 7

Measure 8

Figure 6. Pictorial Model of Reliability [Incidentally, the arrows are backwards in the Straub et al. (2004) article]

20

6b. Reliability
Reliability is all about the consistency of scale items with each other within the same construct (reflective scales only); it has nothing to do with other constructs. A reliable reflective scale is one that has items that are highly correlated with each other (demonstrates high multicollinearity). The essential scale reliability questions are: 1. Is the Cronbachs alpha higher than .6 for exploratory scales or .7 for confirmatory scales? See Nunnally, 1967. [Nunnally, J.C. 1967. 2. Can we improve the reliability by dropping items? This is a tradeoff in that higher number of items almost always yields a higher Cronbachs alpha, but respondents get tired when the questionnaire is too long.
Psychometric Theory. New York: New York.]

21

6b. Reliability
Demonstration exercise using SPSS and TAMAVG.sav dataset

22

6c. Construct validity

Latent Reflective Construct C

Notice anything wrong with this research model?

Measure 7

Measure 8

Latent Reflective Construct A

Latent Formative Construct B

Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 6

Figure 3. Pictorial Model of Construct Validity

23

6c. Construct validity

Construct validity is all about the stickiness of the measures that have been chosen.

Construct validity means that scales items that are supposed to be related to each other DO relate to each other (convergent validity) and those that are NOT supposed to be related to each other do NOT relate to each other. The essential factorial validity questions are: 1. In the presence of other items related to different constructs, do scale items that we expect to be related to each load on the same factor at high levels? (Convergent validity) 2. In the presence of other items related to different constructs, do scale items that we do not expect to relate to each load onto different factors? (Discriminant validity)

24

6c. Construct validity

Notice anything wrong with this research model?
Latent Reflective Construct C

Comparison for stage 2 if you only have one DV?

.Control variables like firm size; demographics like age of respondent or work experience of subjectany construct that should NOT correlate with the DV!!

Measure 7

Measure 8

Latent Reflective Construct A

Latent Formative Construct B

Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 6

Stage 1

Stage 2
25

6c. Construct validity

Demonstration exercise using SPSS and TAMAVG.sav dataset

26

6c. Construct validity

Demonstration exercise using SPSS and TAMAVG.sav dataset

27

7.

Why we should care about having valid instruments

1. GIGO 2. Standards are generally rising across the social sciences and one could be viewed negatively (i.e., in the backwaters) if one is not aware of and practicing these validations. 3. It could be personally and professionally embarrassing if an article is published, but colleagues begin to doubt the worth of the article because the data was never certified as measurement-error controlled (or at least accounted for).

28

Thank you! Any Questions?

29