Principal Component Analysis

Philosophy of PCA
Introduced by Pearson (1901) and
Hotelling (1933) to describe the
variation in a set of multivariate data in
terms of a set of uncorrelated variables
We typically have a data matrix of n
observations on p correlated variables
x1,x2,xp
PCA looks for a transformation of the xi
into p new variables yi that are
uncorrelated
The data matrix
case ht (x1) wt(x2) age(x3) sbp(x4) heart rate (x5)
1 175 1225 25 117 56
2 156 1050 31 122 63
n 202 1350 58 154 67
Reduce dimension

The simplet way is to keep one

variable and discard all others: not
reasonable!
Wheigt all variable equally: not
reasonable (unless they have same
variance)
Wheigted average based on some
citerion.
Which criterion?
Let us write it first

Looking for a transformation of the data

matrix X (nxp) such that

Y= T X=1 X1+ 2 X2+..+ p Xp

Where =(1 , 2 ,.., p)T is a column vector

of wheights with

1+ 2+..+ p =1
One good criterion

Maximize the variance of the projection of

the observations on the Y variables
Find so that

Var(T X)= T Var(X) is maximal

The matrix C=Var(X) is the covariance

matrix of the Xi variables
Let us see it on a figure
Good Better
Covariance matrix

v( x1 ) c(x1,x2 ) ........c(x1,x p )

C=
c(x1,x2 ) v( x2 ) ........c(x2 ,x p )


c(x ,x ) c(x ,x )..........v( x )
1 p 2 p p
And so.. We find that
The direction of is given by the
eigenvector 1 correponding to the
largest eigenvalue of matrix C
The second vector that is orthogonal
(uncorrelated) to the first is the one
that has the second highest variance
which comes to be the eignevector
corresponding to the second
eigenvalue
And so on
So PCA gives

New variables Yi that are linear

combination of the original variables
(xi):
Yi= ai1x1+ai2x2+aipxp ; i=1..p
The new variables Yi are derived in
decreasing order of importance;
they are called principal components
Calculating eignevalues and
eigenvectors
The eigenvalues i are found by
solving the equation
det(C-I)=0
Eigenvectors are columns of the
matrix A such that
1 0 ........0
C=A D A T
0 2 .......0
Where D= 0

0 ............
p
An example

Let us take two variables with covariance c>0

C= 1 c C-I= 1 c
c 1

c 1

det(C-I)=(1- )-c

Solving this we find 1 =1+c

2 =1-c < 1
and eigenvectors

Any eigenvector A satisfies the condition

CA=A

a1 1 c a1 a1 ca2 a1

A= CA= = =

2
a
c 1 a2 ca1 a2 a2

Solving we find A1 A2
PCA is sensitive to scale

If you multiply one variable by a

scalar you get different results
(can you show it?)
This is because it uses covariance
matrix (and not correlation)
PCA should be applied on data that
have approximately the same scale
in each variable
Interpretation of PCA

The new variables (PCs) have a

variance equal to their corresponding
eigenvalue
Var(Yi)= i for all i=1p
Small i small variance data
change little in the direction of
component Yi
The relative variance explained by
each PC is given by i / i
How many components to keep?

Enough PCs to have a cumulative

variance explained by the PCs that is
>50-70%
Kaiser criterion: keep PCs with
eigenvalues >1
Scree plot: represents the ability of
PCs to explain de variation in data
Do it graphically
Interpretation of components
See the wheights of variables in each
component
If Y1= 0.89 X1 +0.15X2-0.77X3+0.51X4
Then X1 and X3 have the highest
wheights and so are the mots
important variable in the first PC
See the correlation between variables
Xi and PCs: circle of correlation
Circle of correlation
Normalized (standardized) PCA

If variables have very heterogenous

variances we standardize them
The standardized variables Xi*

Xi*= (Xi-mean)/variance

The new variables all have the same

variance (1), so each variable have
the same wheight.
Application of PCA in Genomics

PCA is useful for finding new, more

informative, uncorrelated features; it
reduces dimensionality by rejecting
low variance features
Analysis of expression data
Analysis of metabolomics data (Ward
et al., 2003)
However

PCA is only powerful if the biological

question is related to the highest
variance in the dataset
If not other techniques are more
useful : Independent Component
Analysis
Introduced by Jutten in 1987
What is ICA?
That looks like that
The idea behind ICA
How it works?
Rationale of ICA
Find the components Si that are as
independent as possible in the sens of
maximizing some function F(s1,s2,.,sk)
that measures indepedence
All ICs (except 1) should be non-
Normal
The variance of all ICs is 1
There is no hierarchy between ICs
How to find ICs ?
Many choices of objective function F
Mutual information
f ( s1 , s2 ,..., sk )
MI f ( s1 , s2 ,..., sk ) Log
f1 ( s1 ) f 2 ( s2 )... f k ( sk )
We use the kurtosis of the variables
to approximate the distribution
function
The number of ICs is chosen by the
user
Difference with PCA
It is not a dimensionality reduction
technique
There is no single (exact) solution for
components; uses different algorithms
(in R: FastICA, PearsonICA, MLICA)
ICs are of course uncorrelated but
also as independent as possible
Uninteresting for Normally distributed
variables
Example: Lee and Batzoglou (2003)

Microarray expression data on 7070

genes in 59 Normal human tissue
samples (19 types)
We are not interested in reducing
dimension but rather in looking for
genes that show tissue specific
expression profile (what make tissue
types differents)
PCA vs ICA

Hsiao et al (2002) applied PCA and

by visual inspection observed three
gene cluster of 425 genes: liver-
specific, brain-specific and muscle-
specific
ICA identified more tissue-specific
genes than PCA