Factor Analysis

Factor Analysis
Factor analysis is a general name denoting a class of procedures primarily
used for data reduction and summarization.
Factor analysis is an interdependence technique in that an entire set of
interdependent relationships is examined without making the distinction
between dependent and independent variables.
Factor analysis is used in the following circumstances:
To identify underlying dimensions, or factors, that explain the
correlations among a set of variables.
To identify a new, smaller, set of uncorrelated variables to replace the
original set of correlated variables in subsequent multivariate analysis
(regression or discriminant analysis).
To identify a smaller set of salient variables from a larger set for use in
subsequent multivariate analysis.

Factor Analysis Model

Mathematically, each variable is expressed as a linear combination
of underlying factors. The covariation among the variables is
described in terms of a small number of common factors plus a
unique factor for each variable. If the variables are standardized,
the factor model may be represented as:
Xi = Ai 1F1 + Ai 2F2 + Ai 3F3 + . . . + AimFm + ViUi
where

Xi
Aij
F
Vi

Ui
m

=
=

i th standardized variable
standardized multiple regression coefficient of
variable i on common factor j
=
common factor
=
standardized regression coefficient of variable i on
unique factor i
=
the unique factor for variable i
=
number of common factors

Factor Analysis Model

The unique factors are uncorrelated with each other and with the common
factors. The common factors themselves can be expressed as linear
combinations of the observed variables.
Fi = Wi1X1 + Wi2X2 + Wi3X3 + . . . + WikXk
where
Fi
Wi
k

=
=
=

estimate of i th factor
weight or factor score coefficient
number of variables

Factor Analysis Model

It is possible to select weights or factor score coefficients so
that the first factor explains the largest portion of the total
variance.
Then a second set of weights can be selected, so that the
second factor accounts for most of the residual variance,
subject to being uncorrelated with the first factor.
This same principle could be applied to selecting additional
weights for the additional factors.

Statistics Associated with Factor Analysis

Bartlett's test of sphericity. This- test statistic used to examine
the hypothesis that the variables are uncorrelated in the
population.
In other words, the population correlation matrix is an identity matrix;
each variable correlates perfectly with itself (r = 1) but has no
correlation with the other variables (r = 0).

Correlation matrix. It is a lower triangle matrix showing the

simple correlations, r, between all possible pairs of variables
included in the analysis. The diagonal elements, which are all
1, are usually omitted.

Statistics Associated with Factor Analysis

Communality. Communality is the amount of variance a variable
shares with all the other variables being considered.
This is also the proportion of variance explained by the common factors.

Eigenvalue. The eigenvalue represents the total variance

explained by each factor.
Factor loadings. Factor loadings are simple correlations between
the variables and the factors.
Factor loading plot. A factor loading plot is a plot of the original
variables using the factor loadings as coordinates.
Factor matrix. A factor matrix contains the factor loadings of all
the variables on all the factors extracted.

Statistics Associated with Factor Analysis

Factor scores. Factor scores are composite scores estimated for each
respondent on the derived factors.

Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy. The KaiserMeyer-Olkin (KMO) measure of sampling adequacy is an index used to examine
the appropriateness of factor analysis. High values (between 0.5 and 1.0)
indicate factor analysis is appropriate. Values below 0.5 imply that factor
analysis may not be appropriate.

Percentage of variance. The percentage of the total variance attributed to

each factor.

Residuals are the differences between the observed correlations, as given in

the input correlation matrix, and the reproduced correlations, as estimated
from the factor matrix.

Scree plot. A scree plot is a plot of the Eigenvalues against the number of
factors in order of extraction.

Conducting Factor Analysis

Problem formulation
Construction of the Correlation Matrix

Method of Factor Analysis

Determination of Number of Factors
Rotation of Factors
Interpretation of Factors

Calculation of
Factor Scores
Determination of Model Fit

Selection of
Surrogate Variables

Conducting Factor Analysis- Formulate the Problem

The objectives of factor analysis should be identified.
The variables to be included in the factor analysis should be
specified based on
past research, theory, and judgment of the researcher.
It is important that the variables be appropriately measured on an interval
or ratio scale.

An appropriate sample size should be used.

As a rough guideline, there should be at least four or five times as many
observations (sample size) as there are variables.
Note: if sample size is small you need to be cautious in interpreting the result

Express your degree of agreement

Items

S
D
A

S
A

It is important to buy a tooth paste that prevents cavity

1 2 3 4 5
Example: benefits consumer speak from purchase of tooth paste
I like a tooth paste that gives shinyAteeth
multi item scale Likert scale 1 2 3 4 5

6 7

A toothpaste should strengthen my gums

1 2 3 4 5

6 7

I prefer the tooth paste that freshens breath

1 2 3 4 5

6 7

Prevention of tooth decay is not an important benefit offered by

the tooth paste

1 2 3 4 5

6 7

The most important consideration in buying a tooth paste is

attractive teeth

1 2 3 4 5

6 7

6 7

Conducting Factor Analysis-Data

RESPONDENT
NUMBER
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

V1
7.00
1.00
6.00
4.00
1.00
6.00
5.00
6.00
3.00
2.00
6.00
2.00
7.00
4.00
1.00
6.00
5.00
7.00
2.00
3.00
1.00
5.00
2.00
4.00
6.00
3.00
4.00
3.00
4.00
2.00

V2
3.00
3.00
2.00
5.00
2.00
3.00
3.00
4.00
4.00
6.00
4.00
3.00
2.00
6.00
3.00
4.00
3.00
3.00
4.00
5.00
3.00
4.00
2.00
6.00
5.00
5.00
4.00
7.00
6.00
3.00

V3
6.00
2.00
7.00
4.00
2.00
6.00
6.00
7.00
2.00
2.00
7.00
1.00
6.00
4.00
2.00
6.00
6.00
7.00
3.00
3.00
2.00
5.00
1.00
4.00
4.00
4.00
7.00
2.00
3.00
2.00

V4
4.00
4.00
4.00
6.00
3.00
4.00
3.00
4.00
3.00
6.00
3.00
4.00
4.00
5.00
2.00
3.00
3.00
4.00
3.00
6.00
3.00
4.00
5.00
6.00
2.00
6.00
2.00
6.00
7.00
4.00

V5
2.00
5.00
1.00
2.00
6.00
2.00
4.00
1.00
6.00
7.00
2.00
5.00
1.00
3.00
6.00
3.00
3.00
1.00
6.00
4.00
5.00
2.00
4.00
4.00
1.00
4.00
2.00
4.00
2.00
7.00

V6
4.00
4.00
3.00
5.00
2.00
4.00
3.00
4.00
3.00
6.00
3.00
4.00
3.00
6.00
4.00
4.00
4.00
4.00
3.00
6.00
3.00
4.00
4.00
7.00
4.00
7.00
5.00
3.00
7.00
2.00

 Using the data given in the slide 12 following

analysis is presented in the following slides
(You can also try using the same data)

Conducting Factor Analysis

Construct the Correlation Matrix

The analytical process is based on a matrix of correlations between the

variables.

Two formal statics for testing appropriateness of factor model:

Bartlett's test of sphericity & Kaiser-Meyer-Olkin (KMO) measure
1. Bartlett's test of sphericity can be used to test the null hypothesis
that the variables are uncorrelated in the population:
in other words, the population correlation matrix is an identity matrix.
(all diagonal terms are 1, & all off diagonal terms are 0)

If this hypothesis cannot be rejected, then the appropriateness of factor

analysis should be questioned.

Conducting Factor Analysis

Construct the Correlation Matrix
Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy.
It compares magnitude of observed correlation coefficients to the
magnitude of the partial correlation coefficient

Value greater than 0.5 is desirable

Note: Small values of the KMO statistic indicate that the correlations
between pairs of variables cannot be explained by other variables and that
factor analysis may not be appropriate.

Correlation Matrix
Variables
V1
V2
V3
V4
V5
V6

V1
1.000
-0.530
0.873
-0.086
-0.858
0.004

V2
0.530
1.000
-0.155
0.572
0.020
0.640

V3
0.873
-0.155
1.000
-0.248
-0.778
-0.018

V4
-0.086
-0.572
-0.248
1.000
-0.007
0.640

V5
0.858
0.020
--0.778
-0.007
1.000
-0.136

V6
0.004
0.640
-0.018
0.640
0.136
1.000

Correlation Matrix
Variables

V1

V2

V3

V4

V5

V1

1.000

V2

-0.530

1.000

V3

0.873

-0.155

1.000

V4

-0.086

0.572

-0.248

1.000

V5

-0.858

0.020

-0.778

-0.007

1.000

V6

0.004

0.640

-0.018

0.640

-0.136

V6

1.000

Conducting Factor Analysis

Determine the Method of Factor Analysis
After determining the factor analysis is suitable for analyzing date
choose the methods:

In principal components analysis

In common factor analysis

Note: Usually principal component method is followed

Conducting Factor Analysis

Determine the Method of Factor Analysis

In principal components analysis, the total variance in the data is

considered.
The diagonal of the correlation matrix consists of unities, and full variance

is brought into the factor matrix.

PCA is recommended when the primary concern is to determine the
minimum number of factors that will account for maximum variance in the
data for use in subsequent multivariate analysis.
The factors are called principal components.

In common factor analysis, the factors are estimated based only on the
common variance.
Communalities are inserted in the diagonal of the correlation matrix.
This method is appropriate when the primary concern is to identify the underlying
dimensions and the common variance is of interest.
This method is also known as principal axis factoring.

What is meant by total variance in the data set? To understand the meaning of
total

variance as it is used in a principal component analysis, remember that the observed

variables are standardized in the course of the analysis. This means that each variable is
transformed so that it has a mean of zero and a variance of one.
The total variance in the data set is simply the sum of the variances of these
observed variables. Because they have been standardized to have a variance of one,
each observed variable contributes one unit of variance to the total variance in the
data set. Because of this, the total variance in a principal component analysis will
always be equal to the number of observed variables being analyzed.
For example:
if seven variables are being analyzed, the total variance will equal seven. The
components that are extracted in the analysis will partition this variance: perhaps the
first component will account for 3.2 units of total variance; perhaps the second
component will account for 2.1 units. The analysis continues in this way until all of the
variance in the data set has been accounted for.

What is a communality?
A communality refers to the percent of variance in an
observed variable that is accounted for by the retained
components (or factors).
A given variable will display a large communality if it
loads heavily on at least one of the studys retained
components.
Although communalities are computed in both
procedures, the concept of variable communality is
more relevant in a factor analysis than in principal
component analysis.

Results of Principal Components Analysis

Communalities
Variables
V1
V2
V3
V4
V5
V6

Initial
1.000
1.000
1.000
1.000
1.000
1.000

Extraction
0.926
0.723
0.894
0.739
0.878
0.790

All are above 0.5

(2.731/6)100=45.52

Initial Eigen values

Factor
1
2
3
4
5
6

Eigen value
2.731
2.218
0.442
0.341
0.183
0.085

% of variance
45.520
36.969
7.360
5.688
3.044
1.420

6.0
decreasing

Cumulat. %
45.520
82.488
89.848
95.536
98.580
100.000

Results of Principal Components Analysis- how many factors to

extract?
Extraction Sums of Squared Loadings
Factor
1
2

Eigen value
2.731
2.218

% of variance
45.520
36.969

Cumulat. %
45.520
82.488

Factor Matrix
Variables
V1
V2
V3
V4
V5
V6

Factor 1
0.928
-0.301
0.936
-0.342
-0.869
-0.177

Factor 2
0.253
0.795
0.131
0.789
-0.351
0.871

Rotation Sums of Squared Loadings

Factor
1
2

Eigenvalue
2.688
2.261

% of variance
44.802
37.687

Cumulat. %
44.802
82.488

Results of Principal Components Analysis

Rotated Factor Matrix
Variables
V1
V2
V3
V4
V5
V6

Factor 1
0.962
-0.057
0.934
-0.098
-0.933
0.083

Factor 2
-0.027
0.848
-0.146
0.845
-0.084
0.885

Factor Score Coefficient Matrix

Variables
V1
V2
V3
V4
V5
V6

Factor 1
0.358
-0.001
0.345
-0.017
-0.350
0.052

Factor 2
0.011
0.375
-0.043
0.377
-0.059
0.395

How are the above factor scores

for each case calculated?
The answer is that an equation
is used where the dependent variable
is the predicted factor score
and the independent variables are
the observed variables.
We can check this but to do this we need
two more pieces of information the factor
score coefficient matrix and the
standardized scores for the
observed variables.

Results of Principal Components Analysis

The lower left triangle contains the reproduced correlation matrix;
the diagonal, the communalities; the upper right triangle, the residuals
between the observed correlations and the reproduced correlations.

Factor Score Coefficient Matrix

Variables
V1
V2
V3
V4
V5
V6

V1
V2
V3
V4
V5
V6
0.926
0.024 -0.029
0.031
0.038 -0.053
-0.078
0.723
0.022 -0.158
0.038 -0.105
0.902 -0.177
0.894 -0.031
0.081
0.033
-0.117
0.730 -0.217
0.739 -0.027 -0.107
-0.895 -0.018 -0.859
0.020
0.878
0.016
0.057
0.746 -0.051
0.748 -0.152
0.790

Conducting Factor Analysis -Determine the Number of Factors

- how many factors to consider

A Priori Determination.
Because of prior knowledge, the researcher may knows how many factors
to expect and thus can specify the number of factors to be extracted
beforehand. ( review of literature /judgment/ focus group )

Determination Based on Eigenvalues.

In this approach, only factors with Eigen values greater than 1.0 are
retained.
An Eigen value represents the amount of variance associated with the factor.
Hence, only factors with a variance greater than 1.0 are included.
Factors with variance less than 1.0 are no better than a single variable, since, due to
standardization, each variable has a variance of 1.0.

If the number of variables is less than 20, this approach will result in a
conservative number of factors.

Conducting Factor Analysis

Determine the Number of Factors

Determination Based on Scree Plot.

It is a plot of the Eigenvalues against the number of factors in
order of extraction.
Experimental evidence indicates that the point at which the scree begins
denotes the true number of factors.
Generally, the number of factors determined by a scree plot will be one or
a few more than that determined by the Eigenvalue criterion.

Determination Based on Percentage of Variance.

In this approach the number of factors extracted is determined
so that the cumulative percentage of variance extracted by the
factors reaches a satisfactory level.
It is recommended that the factors extracted should account for at least
60% of the variance.

Scree Plot
3.0

2.5

Eigenvalue

2.0
1.5
1.0
0.5
0.0
1

3
4
5
Component Number

Conducting Factor Analysis

Determine the Number of Factors

Determination Based on Split-Half Reliability.

The sample is split in half and factor analysis is performed on
each half.
Only factors with high correspondence of factor loadings across
the two subsamples are retained.
Determination Based on Significance Tests.
It is possible to determine the statistical significance of the
separate Eigenvalues & retain only those factors that are
statistically significant.
A drawback is that with large samples (size greater than 200), many
factors are likely to be statistically significant, although from a practical
viewpoint many of these account for only a small proportion of the total
variance.

Out put
Important out put of the factor analysis is Factor Matrix, also called
as factor pattern matrix
it contains the coefficients used to express the standardized
variables in terms of the factors
these coefficients and factor loadings represents the
correlations between the factors and variables
Coefficients with a large absolute value indicates that the
factor & variable are closely related
coefficients of factor matrix is used to interpret the factor

Conducting Factor Analysis- Rotate Factors

Although the initial / un-rotated factor matrix indicates the
relationship between the factors and individual variables, it
seldom results in factors that can be interpreted
because the factors are correlated with many variables.

Therefore, through rotation the factor matrix is transformed into a

simpler one that is easier to interpret.
Variable

High loadings before rotation

2
X

Variable

2
3

2
X

X
X

High loadings after rotation

Conducting Factor Analysis

Rotate Factors
In rotating the factors, we would like each factor to have
nonzero, or significant, loadings or coefficients for only some of
the variables.
Likewise, we would like each variable to have nonzero or
significant loadings with only a few factors, if possible with only
one.

Conducting Factor Analysis - Rotate Factors

1.The rotation is called orthogonal rotation if the axes are
maintained at right angles.
The most commonly used method for rotation is the varimax procedure.
This is an orthogonal method of rotation that minimizes the number of variables with
high loadings on a factor, thereby enhancing the interpretability of the factors.
Orthogonal rotation results in factors that are uncorrelated.

2. The rotation is called oblique rotation when the axes are not
maintained at right angles, and the factors are correlated.
Sometimes, allowing for correlations among factors can simplify the factor pattern matrix.

Oblique rotation should be used when factors in the population are likely to be strongly
correlated.

Orthogonal Factor Rotation

Oblique Factor Rotation

Conducting Factor Analysis - Rotate Factors

Rotation achieves simplicity & enhances interpretability:
Though the rotation does not affect the communalities &
percentage of total variance explained
Loading of variable get restructured
Variance explained by the individual factor is redistributed by
rotation
Percentage of variance accounted for by each factor does not
change
Variables do not correlates highly on many factors

Conducting Factor Analysis-Interpret Factors

A factor can then be interpreted in terms of the variables that
load high on it.
Another useful aid in interpretation is to plot the variables, using
the factor loadings as coordinates.
Variables at the end of an axis are those that have high loadings
on only that factor, and hence describe the factor.

Factor Loading Plot

Rotated Component Matrix
Variable

Component
2

Component Plot in Rotated Space

Component 1

V1

V6

V2

V4

V2

1.0

0.0

V3
V1

V5

Component 2

0.5

-0.5
-1.0
1.0

0.5

0.0

-0.5

-1.0

0.962
-5.72E-02

-2.66E-02
0.848

V3

0.934

-0.146

V4

-9.83E-02 0.854

V5

-0.933

V6

8.337E-02 0.885

-8.40E-02

Conducting Factor Analysis

Calculate Factor Scores
It is essential to calculate factor score for each respondent, if
researchers likes to consider composite variable for
multivariate analysis.
The factor scores for the I th factor may be estimated
as follows:

Fi = Wi1 X1 + Wi2 X2 + Wi3 X3 + . . . + Wik Xk

Conducting Factor Analysis- Select Surrogate Variables

By examining the factor matrix, one could select for each factor
the variable with the highest loading on that factor.
That variable could then be used as a surrogate variable for the
associated factor.
However, the choice is not as easy if two or more variables have
similarly high loadings.
In such a case, the choice between these variables should be based on
theoretical and measurement considerations.

Conducting Factor Analysis- Determine the Model Fit

Determination of model fit is the final step in factor analysis
Assumption: observed correlation between variables can be
attributed to common factors
The correlations between the variables can be deduced or
reproduced from the estimated correlations between the
variables and the factors.
The differences between the observed correlations (as given in
the input correlation matrix) and the reproduced correlations
(as estimated from the factor matrix) can be examined to
determine model fit.
These differences are called residuals.

SPSS Windows
To select this procedures using SPSS for Windows click:
Analyze>Data Reduction>Factor

Factor Analysis Result (Data Response to SPSS and computer )

Factor Analysis-result
Result of un rotated factor analysis
Data is - anxiety about SPSS

Factor Analysis-result
Result of un rotated factor analysis
KMO and Bartlett's Test
Kaiser-Meyer-Olkin Measure of Sampling Adequacy.

Bartlett's Test of Sphericity

0.9302

Approx. ChiSquare
df
Sig.

19334
253
0

Factor Analysis-result- un rotated

Total Variance Explained
Component

Initial Eigenvalues
Extraction Sums of Squared Loadings
% of
Varianc Cumul
Total e
ative % Total
% of Variance
Cumulative %
1 7.29 31.696 31.696
7.29
31.6958568
31.6958568
2 1.739 7.5601 39.256 1.739
7.560124986
39.25598178
3 1.317 5.725 44.981 1.317
5.725006643
44.98098843
4 1.227 5.3356 50.317 1.227
5.335644146
50.31663257
5 0.988 4.2951 54.612
6 0.895 3.8927 58.504
7 0.806 3.5024 62.007
8 0.783 3.4036 65.41
9 0.751 3.2651 68.676
10 0.717 3.1172 71.793
11 0.684 2.9721 74.765
12 0.67 2.9109 77.676
13 0.612 2.6609 80.337
14 0.578 2.5119 82.849
15 0.549 2.3878 85.236
16 0.523 2.2746 87.511
17 0.508 2.2104 89.721
18 0.456 1.9823 91.704
19 0.424 1.8426 93.546
20 0.408 1.773 95.319
21 0.379 1.6499 96.969
22 0.364 1.5827 98.552

Factor analysis output

Factor-1

Items
Loadings
I have little experience of computers
0.80
All computers hate me
0.64
Computers are useful only for playing games
0.55
I worry that I will cause irreparable damage
because of my in-competenece with computers 0.65
Computers have minds of their own and
deliberately go wrong whenever I use them
0.58
Computers are out to get me
0.46
SPSS always crashes when I try to use it
0.68
Reliability Statistics
Cronbach's Alpha
.674

Factor-2

Items
Statistics makes me cry
Standard deviations excite me
I dream that Pearson is attacking me with
correlation coefficients
I don't understand statistics
People try to tell you that SPSS makes
statistics easier to understand but it doesn't
I weep openly at the mention of central
tendency
I can't sleep for thoughts of eigen vectors
I wake up under my duvet thinking that I am
trapped under a normal distribution
Reliability Statistics
Cronbach's Alpha
.605

Loadings
0.50
0.57
0.52
0.43
0.52
0.51
0.68
0.66

Factor-3

Items

Loadings

I have never been good at mathematics

0.83

I did badly at mathematics at school

0.75

I slip into a coma whenever I see an

equation

0.75

Reliability Statistics
Cronbach's Alpha
.819

Factor-4

Items
My friends will think I'm stupid for not being able to cope
with SPSS

Loadings
0.54

My friends are better at statistics than me

0.65

Everybody looks at me when I use SPSS

0.43

My friends are better at SPSS than I am

0.65

If I'm good at statistics my friends will think I'm a nerd

0.59

Reliability Statistics
Cronbach's Alpha
.570

Factor Analysis-result
Result of rotated factor analysis
Run the factor analysis with out considering the
variables having extraction value less than .4

Note: Items 5,10 15 and 19 are deleted and rotated

factor analysis result is reported below
You can observe that variance explained improved ,
and factor membership also changed

KMO and Bartlett's Test

Kaiser-Meyer-Olkin Measure of Sampling Adequacy.
.918
Bartlett's Test of Sphericity Approx. Chi-Square 16263.271
df
171
Sig.
.000

Factor analysis output

Factor-1

Items

Loadings

I have little experience of computers

0.80

All computers hate me

0.68

My friends are better at statistics than me

People try to tell you that SPSS makes statistics easier to
understand but it doesn't
I worry that I will cause irreparable damage because of
my incompetenece with computers
Computers have minds of their own and deliberately go
wrong whenever I use them

0.07

SPSS always crashes when I try to use it

0.74

Reliability Statistics
Cronbach's Alpha = .710

0.53
0.68
0.62

Factor-2

Items

Loadings

Statiscs makes me cry

0.44

Standard deviations excite me

I dream that Pearson is attacking me with correlation
coefficients

0.58

I weep openly at the mention of central tendency

0.51

I can't sleep for thoughts of eigen vectors

I wake up under my duvet thinking that I am trapped
under a normal distribtion

0.70

Reliability Statistics
Cronbach's Alpha = .391

0.46

0.62

Factor-3

Items

Loadings

I have never been good at mathematics

0.85

I did badly at mathematics at school

0.76

I slip into a coma whenever I see an

equation

0.76

Reliability Statistics
Cronbach's Alpha =

.819

Factor-4

Items
My friends will think I'm stupid for not being able to cope
with SPSS

0.51

My friends are better at SPSS than I am

0.67

If I'm good at statistics my friends will think I'm a nerd

0.64

Reliability Statistics
Cronbach's Alpha = .409

Note: Cronbachs Alpha Is less than .6

Loadings