Multiple Linear Regression

Sasadhar Bera, IIM Ranchi

Multiple Linear Regression Model

Multiple linear regression involves one dependent
variable and more than one independent variable. The
equation that describes multiple linear regression model is
given below:
y = 0 + 1 x1 + 2 x2 + .

. + k xk +

y is dependent variable and x1, x2 , .

.
.,xk are
independent variables. These independent variables being
used to predict the dependent variable.

0 , 1 , 2 , . . ., k are total (k+1) unknown regression

coefficients (also called model parameters). These
regression coefficients are estimated based on observed
sample data.
The term (pronounced as epsilon) is random error.
Sasadhar Bera, IIM Ranchi

Data for Multiple Regression

Suppose that n number of observations are collected for
response variable (y) and k number of independent
variables present in the regression model.
i = 1, 2, . . ., n
y
y1
y2
.
yi
.
yn

x1
x11
x21
.
xi1
.
xn1

j = 1, 2, . . .,k
x2
x12
x22
.
xi2
.
xn2

.
.
.
.
.
.
.

xj
x1j
x2j
.
xij
.
xnj

.
.
.
.
.
.
.

xk
x1k
x2k
.
xik
.
xnk

Sasadhar Bera, IIM Ranchi

Scalar Notation: Multiple Linear Regression

Suppose that n number of observations are collected for
response variable (y) and k number of independent
variables present in the regression model.
The scalar notation of regression model:

yi = 0 + 1 xi1 + 2 xi2 + .
i = 1, 2, . . ., n
j = 1, 2, . . .,k

. + j xij + . . + k xik + i

n = total number of observations

k = number of independent variables

j s are model parameters.

Sasadhar Bera, IIM Ranchi

Matrix Notation: Multiple Linear Regression

Suppose that n number of observations are collected for
response variable (y) and k number of independent
variables present in the regression model.
yn1 = Xn(k+1) (k+1) 1 + n1
n = total number of observations, k = total number of
variables, is model parameters in vector notation.
y1
.

y yi

.
y n

1 x11 . x1 j

. . .
.
X 1 x i1 . x ij

. . .
.
1 x . x
n1
nj

.
.
.
.
.

x 1k

.
x ik

.
x nk

0

1
.

j
.

k

1
.

i

.
n

Sasadhar Bera, IIM Ranchi

Model Parameter Estimation

The error in regression model is the difference between
actual and predicted value. It may be positive or negative
value.
Error is also known as residual. Predicted value by
regression equation is called fitted value or fit.
The sum of squared difference between the actual and
predicted values known as sum of square of error. Least
square method minimizes the sum of square of error to
find out the best fitting plane.
It is to be noted that the regressor variables in linear
regression model are non-random. That means its values
are fixed.
Sasadhar Bera, IIM Ranchi

Model Parameter Estimation (Contd.)

In matrix notation, the regression equation:
y =X +

By using least square estimator, we want estimate

n

that minimizes L =

i 1

2
i

=
T

y X ( y X)
T

The least square estimator must satisfy:

T
T
( L) 2 X y 2 X X 0

( XT X)1 XT y , estimated model parameters.

The fitted regression line: y X

Sasadhar Bera, IIM Ranchi

Estimated Residual and Standard Error

For

ith

observation (Xi), predicted value or Fit :

y i Xi

Error in the fit called residual:

ei y i y i
n

2
e
i

Mean Square Error = MSE =

i 1

n k 1

where n is the total number of observations, k is number

of regressors.

Standard error (SE) of estimate = =

MSE

Variance( ) = (X T X) 1
Sasadhar Bera, IIM Ranchi

Testing Significance of Regression Model

The test for significance of regression is a test to
determine if there is a linear relationship between the
response variable and regressor variables.
H0 : 1 = 2 = . . . = k = 0
H1 : At least one j is not zero
The test procedure involves an analysis of variance
(ANOVA) partitioning of the total sum of square into a sum
of squares due to regression and a sum of square due to
error (or residual)
Total number of model parameters = p = Number of
regression coefficients = (k+1)
Sasadhar Bera, IIM Ranchi

10

Testing Significance of Regression Model (Contd.)

ANOVA table
Source of
Variation
Regression
Residual
error
Total

DF

SS

MS

FCal

SSR

SSR /k =MSR

MSR/MSE

n k-1

SSE

SSE / (n-k-1)
= MSE

n 1

TSS

y
2
i
n

T
SSR yi y XT y i1

n
i 1
n

TSS = SSR + SSE

SSE yi yi y T y XT y

i 1
n

TSS yi y
i 1

Sasadhar Bera, IIM Ranchi

11

Significance Test of Individual Regression

Coefficient
Adding an unimportant variable to the model can actually
increase the mean square error, thereby decreasing the
usefulness of the model.
The hypothesis for testing the significance of any
individual regression coefficient, say j is
H0: j = 0

H1: j 0

Test Statistic = Tcal =

j
2 C jj

, ( n k 1)

where 2 is mean square error (MSE) and C is the diagonal

element of (XTX)-1 . Reject H0 if Tcal > t , ( n k 1)
2

12

Sasadhar Bera, IIM Ranchi

Confidence Interval of Mean Response

In matrix notation, the regression equation:
y =X +
where Normal (0, 2)

Mean response at a point x0 = [1, x01, x02, . .,x0j, . . .,x0k ]T

Mean response = y = E(y) = E(X ) + E() = X + 0

y|x = E(y | x0 ) = x0
0

var(y | x0 )

x T0 (XT X)1 x 0

(1-) % confidence interval of mean response at point x0

y|x

( n p )

x T0 (XT X)1 x 0
Sasadhar Bera, IIM Ranchi

13

Coefficient of Multiple Determination

Coefficient of multiple determination =

R2

SSR
=
TSS

SSR TSS SSE

SSE

1
TSS
TSS
TSS
SSR : Sum of square due to regression
SSE : Sum of square due to error
TSS : Total sum of square

Coefficient of variation is the fraction of variation of the

dependent variable explained by regressor variables.
R2 is measure the goodness of linear fit. The better the
linear fit is, the R2 closer to 1.
14

Sasadhar Bera, IIM Ranchi

Coefficient of Multiple Determination (Contd.)

The major drawback of using coefficient of multiple
determination (R2) is that adding a predictor variable to the
model will always increase R2, regardless of whether the
additional variable is significant or not. To avoid such
situation, regression model builders prefer to use adjusted
R2 statistic.

SSE

2
adj

n 1
( n p)
(1 R 2 )
1
1
TSS
n p

(n 1)

In general, adjusted R2 statistic will not increase as variables

are added to the model.

When R2 and adjusted R2 differ dramatically there is a good

chance that non-significant terms have been included in the
15
model.
Sasadhar Bera, IIM Ranchi