11

Statistical Inferences

Once we have estimated the unknown parameters that appear in an algebraic

or ODE model, it is quite important to perform a few additional calculations to establish estimates of the standard error in the parameters and in the expected re-

sponse variables. These additional computational steps are very valuable as they provide us with a quantitative measure of the quality of the overall fit and inform us how trustworthy the parameter estimates are.
11.1 INFERENCES ON THE PARAMETERS

When the Gauss-Newton method is used to estimate the unknown parameters, we linearize the model equations and at each iteration we solve the corresponding linear least squares problem. As a result, the estimated parameter values have linear least squares properties. Namely, the parameter estimates are normally distributed, unbiased (i.e., E(k*)=k) and their covariance matrix is given by COV(k*) = Og [A*]~' (11.1)

where A* is matrix A evaluated at k*. It should be noted that for linear least
squares matrix A is independent of the parameters while this is clearly not the case

for nonlinear least squares problems. The required estimate of the variance Og is obtained from 2 S(k*) S(k*) (11.2)
(d.f.) N m- p
177

Copyright 2001 by Taylor & Francis Group, LLC

178

Chapter 11

where (d.f.)= Nmp are the degrees of freedom, namely the total number of measurements minus the number of unknown parameters.
The above expressions for the C(3F(k*) and ojr are valid, if the statistically

correct choice of the weighting matrix Q;, (i=l,...,N) is used in the formulation of the problem. Namely, if the errors in the response variables (EJ, i=l,...,N) are normally distributed with zero mean and covariance matrix, COF(]) = a;: M,
(1L3)

we should use [Mj]~ 1-1 as the weighting matrix Q. where the matrices M j; i=l,...,N
are known whereas the scaling factor, a^ , could be unknown. Based on the structure of MJ we arrive at the various cases of least squares estimation (Simple LS, Weighted LS or Generalized LS) as described in detail in Chapter 2. Although the computation of COK(k*) is a simple extra step after convergence of the Gauss-Newton method, we are not obliged to use the Gauss-Newton method for the search of the best parameter values. Once k* has been obtained using any search method, one can proceed and compute the sensitivity coefficients by setting up matrix A and thus quantify the uncertainty in the estimated parameter values by estimating COK(k*). Approximate inference regions for nonlinear models are defined by analogy to the linear models. In particular, the (1 -a) 100% joint confidence region for the parameter vector k is described by the ellipsoid,

or
[ k - k f [A*r 1 [k-k*] = N m- p F Nm P (1Mb)

where a is the selected probability level in Fisher's F-distribution and ^Nm-p is obtained from the F-distribution tables with Vj=p and v2=(N/w-/?) degrees of freedom. The corresponding (I -a) 100% marginal confidence interval for each parameter, kj, i=l,2,...,p, is given by k * - t * / 2 o k i < kj <k* +t*/2oki (11.5)

where t^ 7 2 is obtained from the tables of Student's T-distribution with v=(Nw-p) degrees of freedom. The standard error of parameter kj, a k . , is obtained as the
Copyright 2001 by Taylor & Francis Group, LLC

Statistical Inferences

179

square root of the corresponding diagonal element of the inverse of matrix A* multiplied by 6 g , i.e.,

kj

= E

It is reminded that for v>30 the approximation t^ / 2 ~ z^ can be used where zaj2 is obtained from the standard normal distribution tables. Simply put, when we have many data points, we can simply take as the 95% confidence interval twice the standard error (recall that zom$=\ .96 whereas t 0 025=2.042). The linear approximation employed by the Gauss-Newton method in solving the nonlinear least squares problem enables us to obtain inference regions for the parameters very easily. However, these regions are only approximate and practice has shown that in many cases they can be very misleading (Watts and Bates, 1988). Nonetheless, even if the regions are not exact, we obtain a very good idea of the correlation among the parameters. We can compute the exact (1 -a) 100% joint parameter likelihood region using the equation given below
S(k) = S(k*) 1h + ^ _ _ ^ N Nw-_p H

(11.7)

The computation of the above surface in the parameter space is not trivial. For the two-parameter case (p=2), the joint confidence region on the k r k 2 plane can be determined by using any contouring method. The contour line is approximated from many function evaluations of S(k) over a dense grid of (k b k2) values.
11.2 INFERENCES ON THE EXPECTED RESPONSE VARIABLES

Having determined the uncertainty in the parameter estimates, we can proceed and obtain confidence intervals for the expected mean response. Let us first consider models described by a set of nonlinear algebraic equations, y=f(x,k). The 100(1-a)% confidence interval of the expected mean response of the variable y. at x0 is given by
f x

( o,k*)-C2<i

< u

< f ( x 0 , k * ) + tj; / 2 a

(11.8)

Copyright 2001 by Taylor & Francis Group, LLC

180

Chapter 11

where t^ / 2 is obtained from the tables of Student's T-distribution with v=(Nw-p)

degrees of freedom. Based on the linear approximation of the model equations, we have
of1

y0(k) = f(x0,k*)

[k - k *]

(11.9)

5k

with the partial derivative (5fT/5k)r evaluated at x0 and k*. Taking variances from both sides we have
\T

(ll.lOa)

5k

dk

Substitution of the expression for COK(k*) yields,

5f
T

\T

(ll.lOb)
5k

5k

The standard prediction error of yjo, o y .Q , is the square root of the jth diagonal element of COF(y0), namely,

5k

5k

Equation 11.8 represents the confidence interval for the mean expected response rather than a. future observation (future measurement) of the response vari-

able, y 0 . In this case, besides the uncertainty in the estimated parameters, we must
include the uncertainty due to the measurement error (EO). The (l~a)100% confidence interval of y JQ is
(II.12)

where the standard prediction error of y JQ is given by

Copyright 2001 by Taylor & Francis Group, LLC

Statistical Inferences

181

y.,o

-'
V

5f

,1
)

ok

[Af

'*>\
V

(11.13)

ok

Next let us turn our attention to models described by a set of ordinary differential equations. We are interested in establishing confidence intervals for each of the response variables y j , j=l,...,w at any time t=t0. The linear approximation of the output vector at time to, y(t 0 ,k) = Cx(t 0 ,k*) + CG(t 0 )[k-k*] yields the expression for COK(y(t0)),
CW(y 0 ) = CG(t0)COF(k*)CTGT(t0) (11.15) (11.14)

which can be rewritten as

CW(y 0 ) = d2 (11.16)

with the sensitivity coefficients matrix G(to) evaluated at k*. The estimated standard prediction error of yj(to) is obtained as the square root of the j* diagonal element of CO F(y(t0)). A*]- 1 C T G T (t 0 )J i j (11.17)

Based on the latter, we can compute the (l-a)100% confidence interval of the expected mean response of y, at t=t0,
y j (t 0 ,k*)-t^ / 2 d

< u

< y ( t 0 , k * ) + C26

(11.18)

If on the other hand we wish to compute lhe(l-a)100% confidence interval of the response of y, at t=t0, we must include the error term (EO) in the calculation of the standard error, namely we have
v y j (t 0 ,k*)-t a / 2 o y j o

<

y j (t 0 )

<

(11.19)

with
Copyright 2001 by Taylor & Francis Group, LLC

182

Chapter 11

+ {CG(t 0 )[A*]- 1 C T G T (t 0 )J

(11.20)

11.3

MODEL ADEQUACY TESTS

There is a plethora of model adequacy tests that the user can employ to decide whether the assumed mathematical model is indeed adequate. Generally speaking these tests are based on the comparison of the experimental error variance estimated by the model to that obtained experimentally or through other means.
1 1 .3. 1 Single Response Models

Let us first consider models that have only one measured variable (m=\). We shall consider two cases. One where we know precisely the value of the experimental error variance and the other when we have an estimate of it. Namely, there is quantifiable uncertainty in our estimate of the experimental error variance.
CASE 1 : <r is known precisely:

In this case we assume that we know precisely the value of the standard experimental error in the measurements (oe). Using Equation 1 1.2 we obtain an estimate of the experimental error variance under the assumption that the model is adequate. Therefore, to test whether the model is adequate we simply need to test
the hypothesis
H :

model

at any desirable level of significance, e.g., a=0.05. Here with o , , we denote the error variance estimated by the model equations (Equation 11.2); namely, dg is an estimate of Since o g is known exactly (i.e., there is no uncertainty in its value, it is a given number) the above hypothesis test is done through a x -test. Namely,
If Xdata > Xv=(Nm-p),l-a
=>

Reject H0

where
Copyright 2001 by Taylor & Francis Group, LLC

Statistical Inferences

183

and Xv=(Nm-p),l-a 's obtained from the tables of the jf-distribution with degrees

of freedom v=(N/w-p).

CASE 2: o is known approximately:

Let us assume that o is not known exactly, however, we have performed n repeated measurements of the response variable. From this small sample of multiple measurements we can determine the sample mean and sample variance. If s g is the sample estimate of oe , estimated from the n repeated measurements it is given by

\ ^ S7E = / ( y j - y )j
n-1

(11.22)

where the sample mean is obtained from

(11.23)

Again, we test the hypothesis at any desirable level of significance, for example ce=0.05
H :

<* model

In this case, since o g is known only approximately, the above hypothesis is tested using an F-test, i.e.,
i f If r -^ i- l=( F data > Fj_'a
v Nm n w v ~P)> ' 22= ~l

^ =>

n * n Reject H o

where
Copyright 2001 by Taylor & Francis Group, LLC

184

Chapter 11

(H-24)

and F v j = d f >v ' 2 = n ~' is obtained from the tables of the F-distribution.

11.3.2

Multivariate Models

Let us now consider models that have only more than one measured variable (m>\). The previously described model adequacy tests have multivariate extensions that can be found in several advanced statistics textbooks. For example, the book Introduction to Applied Multivariate Statistics by Srivastava and Carter (1983) presents several tests on covariance matrices. In many engineering applications, however, we can easily reduce the problem to the univariate tests presented in the previous section by assuming that the covariance matrix of the errors can be written as CWfe) = cjgM, ; i=l,...,N (11.25)

where M; are known matrices. Actually quite often we can further assume that the matrices IM,, i=l,...,N are the same and equal to matrix M. An independent estimate of COV(z), E , that is required for the adequacy tests can be obtained by performing NK repeated experiments as

or for the case of univariate tests, s , the sample estimate of O E in Equation 11.25, can be obtained from
NR

(N R -i " ' ""

"J

"'

"'

(11.27)

Copyright 2001 by Taylor & Francis Group, LLC