TECHNICAL PAPERS

Risk Mitigation Strategies

for Compliance Testing
Jonathan Harben and Paul Reese
Abstract: Many strategies for risk mitigation are now practiced in calibration laboratories. This paper presents a modern look
at these strategies in terms of compliance to ANSL/NCSLI and ISO standards. It distinguishes between Bench Level and
Program Level risk analysis techniques, which each answer different questions about risk mitigation. It investigates concepts
including the test uncertainty ratio (TUR) and end of period reliability (EOPR) that are directly related to risk, as well as the mathematical boundary conditions of false accept risk to gain a comprehensive understanding of practical, efficient risk mitigation.
The paper presents practices and principals that can allow a calibration laboratory to meet the demand of customers and manage
risk for multifunction instrumentation, while complying with national and international standards.
1. Background

Calibration is all about confidence. In some scenarios, it is important

to have confidence that the certified value of a laboratory reference
standard is within its assigned uncertainty limits. In other scenarios,
confidence that an instrument is performing within its published accuracy specifications may be desired. Confidence in an instrument is
often obtained through compliance testing, which is sometimes called
conformance testing, tolerance testing, or verification testing. For
these types of tests, a variety of strategies have historically been used
to manage the risk of falsely accepting non-conforming items and erroneously passing them as good. This type of risk is called false
accept risk (also known as FAR, probability of false accept (PFA),
consumers risk, or Type II risk). To mitigate false accept risk, simplistic techniques have often relied upon assumptions or approximations that were not well founded. However, high confidence and low
risk can be achieved without relying on antiquated paradigms or unnecessary computations. For example, there are circumstances where
a documented uncertainty is not necessary to demonstrate that false
accept risk was held below certain boundary conditions. This is a
somewhat novel approach with far-reaching implications in the field
of calibration.
While the importance of uncertainty calculations is acknowledged
for many processes (e.g. reference standards calibrations), it might be
unnecessary during compliance tests when historical reliability data is
available for the unit under test (UUT). Many organizations require
a documented uncertainty statement in order to assert a claim of metrological traceability [1], but the ideas presented here offer evidence
that acceptance decisions can be made with high confidence without
direct knowledge of the uncertainty.
In the simplest terms, when measurement & test equipment
(M&TE) owners send an instrument to the calibration laboratory they
want to know, Is my instrument good or bad? During a compliance
test, M&TE is evaluated using laboratory standards to determine if it
is performing as expected. This performance is compared to specifications or tolerance limits that are requested by the end user or customer. These specifications are often the manufacturers published
38

NCSLI Measure

accuracy1 specifications. The customer is asking for an in-tolerance

or out-of-tolerance decision to be made, which might appear to be
a straightforward request. But exactly what level of assurance does
the customer receive when statements of compliance are issued? Is
simply reporting measurement uncertainty enough? What is the risk
that a statement of compliance is wrong? While alluded to in many
international standards documents, these issues are directly addressed
in ANSI/NCSL Z540.3-2006 [2].
Since its publication, sub-clause 5.3b of the Z540.3 has, understandably, received a disproportionate amount of attention compared
with other sections in the standard [3, 4, 5]. This section represents
a significant change when compared to its predecessor, Z540-1 [6].
Section 5.3b has come to be known by many as The 2 % Rule
and addresses calibrations involving compliance tests. It states:
Where calibrations provide for verification that measurement quantities are within specified tolerances, the probability that incorrect
acceptance decisions (false accept) will result from calibration tests
shall not exceed 2% and shall be documented. Where it is not practicable to estimate this probability, the test uncertainty ratio shall be
equal to or greater than 4:1.
Much can be inferred from these two seemingly innocuous statements. The material related to compliance testing in the ISO 17025
[7] is sparse, as that standard is primarily focused on reporting uncertainties with measurement results, similar to Z540.3 section 5.3a. Perhaps the most significant reference to compliance testing in ISO 17025
is found in section 5.10.4.2 (Calibration Certificates) which states that
When statements of compliance are made, the uncertainty of measurement shall be taken into account. However, practically no guidance in provided regarding the methods that could be implemented to
take the measurement uncertainty into account. The American Asso1
The term accuracy is used throughout this paper to facilitate the classical
concept of uncertainty for a broad audience. It is acknowledged that
the VIM [1] defines accuracy as qualitative term, not quantitative, and that
numerical values should not be associated with it.

www.ncsli.org

TECHNICAL PAPERS

Figure 1. Five possible bench level calibration scenarios.

ciation of Laboratory Accreditation (A2LA)

further clarifies the requirements associated
with this concept in R205 [8]:
When parameters are certified to be within
specified tolerance, the associated uncertainty of the measurement result is properly
taken into account with respect to the tolerance by a documented procedure or policy
established and implemented by the laboratory that defines the decision rules used by
the laboratory for declaring in or out of tolerance conditions.2
Moreover, the VIM [1] has recently added
a new Note 7 to the definition of metrological traceability. This note reiterates that the
International Laboratory Accreditation Cooperation (ILAC) requires a documented mea-

Authors
Jonathan Harben
The Bionetics Corporation
M/S: ISC-6175
Kennedy Space Center, FL 32899
jonathan.p.harben@nasa.gov

Paul Reese
Covidien, Inc.
815 Tek Drive
Crystal Lake, IL 60014
paul.reese@covidien.com

39

NCSLI Measure

surement uncertainty for any and all claims of

metrological traceability. However, simply reporting the uncertainty along with a measurement result may not satisfy customer requirements where compliance tests are desired.
Without having a quantifiable control limit
such as false accept risk, this type of reporting
imparts unknown risks to the customer.
The methods presented in this paper provide assurance that PFA risks are held below a specified maximum permissible value
(2 %) without direct knowledge of the uncertainty. However, they may not satisfy the
strict language of national and international
documents, which appear to contain an implicit requirement to document measurement
uncertainties for all calibrations.
Where compliance tests are involved,
the intent of the uncertainty requirements
may (arguably) be to allow an opportunity
to evaluate the risks associated with pass/
fail compliance decisions. If this is indeed
the intent, then the ideas presented here can
provide the same opportunity for evaluation without direct knowledge of the uncertainty. Because considerable effort is often
required to generate uncertainty statements,
it is suggested that accreditation bodies accept the methods described in this paper as
an alternative solution for compliance testing.

2. Taking the Uncertainty

Into Account

What does it mean to take the uncertainty

into account and why it is necessary? For
an intuitive interpretation, refer to Fig. 1. During a compliance test on the bench, what are
the decision rules if uncertainty is taken into
account? For example, during the calibration,
the UUT might legitimately be observed to be
in-tolerance. However, the observation could
be misleading or wrong as illustrated in Fig. 1.
It is understood that all measurements are
only estimates of the true value of the measurand; this true value cannot be exactly known
due to measurement uncertainty. In scenario
#1, a reading on a laboratory standard voltmeter of 9.98 V can confidently lead to an intolerance decision (pass) for this 10 V UUT
source with negligible risk. This is true due to
sufficiently small uncertainty in the measurement process and the proximity of the measured value to the tolerance limit. Likewise,
a non-compliance decision (fail) resulting
from scenario #5 can also be made with high
confidence, as the measured value of 9.83 V
is clearly out-of-tolerance. However, in scenarios #2, #3, and #4, there is significant risk
that a pass/fail decision will be incorrect.
2

The default decision rule is found in

ILAC-G8:1996 [9], Guidelines on Assessment
and Reporting of Compliance with Specification,
section 2.5. With agreement from the customer,
other decision rules may be used as provided for
in this section of the requirements.

www.ncsli.org

TECHNICAL PAPERS
In scenarios, #2, #3, & #4, this uncertainty makes it possible for the true value of
the measurand to be either in or out of tolerance. Consider scenario #3, where the UUT
was observed at 9.90 V, exactly at the lower
allowable tolerance limit. Under such conditions, there is a 50 % probability that either an
in-tolerance or out-of-tolerance decision will
be incorrect, barring any other information.
In fact, even for standards with the lowest
possible uncertainty, the probability of being
incorrect will remain at 50 % in scenario #33.
This concept of bench level risk is addressed
in several documents [9, 10, 11, 12].
The simple analysis of the individual
measurement results presented above is not
directly consistent with the intent of The
2 % rule in Z540.3, although it still has
application. Until now, our discussion has
dealt exclusively with bench level analysis
of measurement decision risk. That is, risk
was predicated only on knowledge of the
relationship between the UUT tolerance, the
measurement uncertainty, and the observed
measurement result made on-the-bench.
However, the computation of false accept
risk, for strict compliance with the 2 % rule
in Z540.3, does not depend on any particular
measurement, nor does it depend on its proximity to a given UUT tolerance limit. Instead,
the 2 % rule in Z540.3 addresses the risk at
the program level, prior to obtaining a measurement result. To understand both bench
level and program level false accept risk, the
intent underlying the 2 % rule and its relationship to TUR and EOPR4 must be examined.
3. The Answer to Two Different
Questions

False accept risk describes the overall probability of false acceptance when pass/fail
decisions are made. False accept risk can be
interpreted and analyzed at either the bench
level or the program level [4]. Both risk levels are described in ASME Technical Report
B89.7.4.1-2005 [13]. The ASME report refers
to bench level risk mitigation as controlling
Bayesian analysis can result in false accept risk
other than 50 % in such instances, where the a
priori in-tolerance probability (EOPR) of the UUT
is known in addition to the measurement result
and uncertainty.

The subject of measurement decision risk

includes not only the probability of false-accept
(PFA), but the probability of correct accept
(PCA), probability of false reject (PFR) and the
probability of correct reject (PCR). While false
rejects can have significant economic impact to
the calibration lab, the discussion in this paper is
primarily limited to false accept risk.

40

NCSLI Measure

the quality of individual workpieces, while

program level risk strategies are described
as controlling the average quality of workpieces. Bench level risk can be thought of as
an instantaneous liability at the time of measurement, whereas program level risk speaks
more to the average probability that incorrect
acceptance decisions will be made based on
historical data. These two approaches are
related, but result in two answers to two different questions. Meeting a desired quality
objective requires an appropriate answer to
an appropriate question, and ambiguity in the
question itself can lead to different assumptions regarding the meaning of false accept
risk. Many international documents discuss
only the bench level interpretation of risk,
and require an actual measurement result to
be available [9, 10, 11, 12]. These documents
describe the most basic implementation of
bench level risk, where no other pre-measurement state of knowledge exists. They
address the instantaneous false accept risk
associated with an acceptance decision for a
single measured value, without the additional
insight provided by historical data. This most
basic of bench level techniques is sometimes
called the confidence level method. However, if a-priori data exists, a more rigorous type of bench-level analysis is possible
using Bayesian methods. By employing prior
knowledge of reliability data, Bayesian analysis updates or improves the estimate of risk.
The Z540.3 standard, however, was intended to address risk at the program level
[14]. When this standard requires the
probability that incorrect acceptance decisions (false accept) will result from calibration tests shall not exceed 2%.., it might
not be evident which view point is being addressed, the bench level or the program level. The implications of this were significant
enough to prompt NASA to request interpretive guidance from the NCSLI 174 Standards
Writing Committee [15]. It was affirmed that
the 2 % false accept requirement applies to
a population of like calibration sessions or
like measurement processes [14]. As such,
Z540.3 section 5.3b does not directly address
the probability of false accept to any single,
discrete measurement result or individual
workpiece and supports the program level
view of risk prior to, and independent of, any
particular measurement result.
In statistical terms, the 2 % rule refers to
the unconditional probability of false acceptance. In terms of program level risk, false

accept risk describes the overall or average

probability of false acceptance decisions to
the calibration program at large. It does not
represent risk associated with any particular
instrument. The 2 % rule speaks to the following question: Given a historical collection of pass/fail decisions at a particular testpoint for a population of like-instruments (i.e.
where the EOPR and TUR are known), what
is the probability that an incorrect acceptance
decision will be made during an upcoming
test? Note that no measurement results are
provided, and that the question is being asked
before the scheduled measurement is ever
made and the average risk is controlled for
future measurements. Even so, the question
can be answered as long as previous EOPR
data on the UUT population is available, and
if the measurement uncertainty (and thus
TUR) is known. In certain circumstances, it
is also possible to comply with the 2 % rule
by bounding or limiting false accept risk using either:

EOPR data without knowledge of the

measurement uncertainty.
TUR without knowledge of EOPR data.

To understand how this is possible, a

closer look at the relationship between false
accept risk, EOPR, and TUR is helpful.
4. End of Period Reliability (EOPR)

EOPR is the probability of a UUT test-point

being in-tolerance at the end of its normal
calibration interval. It is sometimes known as
in-tolerance probability and is derived from
previous calibrations. In its simplest form,
EOPR can be defined as

EOPR =

Number of in-tolerance results

Total number of calibrations

(1)

If prior knowledge tells us that a significant

number of previous measurements for a population of UUTs were very close to their tolerance limits as-received, it can affect the
false accept risk for an upcoming measurement. Consider Fig. 2 where two different
model UUT voltage sources are scheduled
for calibration, model A and model B. The
five previous calibrations on model As have
shown these units to be highly reliable; see
Group A. Most often, they are well within
their tolerance limits and easily comply with
their specifications. In contrast, previous
model B calibrations have seldom met their
specifications; see Group B. Of the last five
www.ncsli.org

TECHNICAL PAPERS

Figure 2. Previous historical measurement data can influence future false accept risk.

Figure 3. The possibility of a false accept for a measurement result.

calibrations, two model Bs were recorded

as being out-of-tolerance and one of them
was barely-in. Therefore, making an in or
out of tolerance decision will be a precarious judgment call, with a high probability of
making a false accept decision.
In Fig. 3, imagine the measurement result
is not yet shown on the chart. If it was known
ahead of time that this upcoming measure41

NCSLI Measure

ment result would be near the tolerance limit,

it can be seen that a false accept would indeed
be more likely given the uncertainty of the
measurement. The critically important point
is this -- if the historical reliability data indicates that in-tolerance probability (EOPR)
of the UUT is poor (up to a point5), the false
accept risk increases.
The previous scenarios assume familiar-

ity with populations of similar instruments

that are periodically recalibrated. But how
can EOPR be reconciled when viewed from
a new laboratorys perspective? Can a
5

Graphs of EOPR vs. false-accept risk can reveal

a perceived decrease in false-accept risk as the
EOPR drops below certain levels. This is due to
the large number of out-of-tolerance conditions
that lie far outside the UUT tolerance limits. This
is discussed later in this paper.

www.ncsli.org

TECHNICAL PAPERS
new laboratory open its doors for business
and meet the 2 % false accept requirement
of Z540.3 without EOPR data? The answer
is yes. However, the new laboratory must
employ bench level techniques, or techniques
such as boundary condition methods or
guardbanding. Such methods are described
later in this paper. This same logic would apply to an established laboratory that receives
a new, unique instrument to calibrate for the
first time. In the absence of historical data,
other appropriate techniques and/or bench
level methods must be employed.
If EOPR data or in-tolerance probability
is important for calculating risk, several other
questions are raised. For example, how good
must the estimate of EOPR be before program level methods can be used to address
false accept risk for a population of instruments? When is the collection of measurement data complete? What are the rules for
updating EOPR in light of new evidence?
Sharing or exchanging EOPR data between
different laboratories has even been proposed
with varying opinions. Acceptance of this
generally depends upon the consistency of
the calibration procedure used and the laboratory standards employed. The rules used
to establish EOPR data can be subjective
(for example, how many samples are available, are first-time calibrations counted, are
broken instruments included, are late calibrations included, and so on). Instruments can
be grouped together by various classifications, such as model number. For example,
reliability data for the M&TE model and
manufacturer level can be used to conservatively estimate the reliability of the M&TE
test point. This is addressed in compliance
Method 1 & 2 of the Z540.3 Handbook [16].

tion Quality Division during the 1950s in

an attempt to minimize incorrect acceptance
decisions. The origins of the ubiquitous 4:1
TUR [18] assume a 95 % in-tolerance probability for both the measuring device and the
UUT. In those pre-computer days, these assumptions were necessary to ease the computational requirements of risk analysis. Since
then, manufacturers specifications have often been loosely inferred to represent 2 or
95 % confidence for many implementations
of TUR, unless otherwise stated. In other
words, it is assumed that all UUTs will meet
their specifications 95 % of the time (i.e.
EOPR will be 95 %). Even if the calibration
personnel did not realize it, they were relying
on these assumptions to gain any utility out of
the 4:1 TUR. However, is the EOPR for all
M&TE really 95 %? That is, are all manufacturers specifications based on two standard
deviations of the product distribution? If they
are not, then the time-honored 4:1 TUR will
not provide the expected level of protection
for the consumer.
While the spirit of Z540.3 is to move away
from the reliance on TUR altogether, its use is
still permitted if adherence to the 2 % rule is
deemed impracticable. The use of the TUR
is discouraged due to the many assumptions it
relies on for controlling risk. However, given
that the false accept risk computation requires
the collection of EOPR data, the use of TUR
might be perceived as an easy way for labs
to circumvent the 2 % rule. Section 3.11 in
Z540.3 redefines TUR as:

5. Test Uncertainty Ratio

At first, this definition appears to be similar to older definitions of TUR. The definition implies that if the numerator, associated
with the specification of the UUT, is a plusor-minus () tolerance, the entire span of the
tolerance must be included. However, this is
countered by the requirement to multiply the
95 % expanded uncertainty of the measurement process in the denominator by a factor
of two. The confidence level associated with
the UUT tolerance is undefined. This quandary is not new, as assumptions about the level
of confidence associated with the UUT (numerator) have been made for decades.
There is, however, a distinct difference
between the TUR as defined in Z540.3 and

It has been shown that EOPR can affect the

false accept risk of calibration processes.
However, test uncertainty ratio (TUR) is likely to be more familiar than EOPR as a metric
of the quality of calibration. The preceding examples show that a lower uncertainty
generally reduces the likelihood of a false
accept decision. The TUR has historically
been viewed as the uncertainty or tolerance
of the UUT in the numerator divided by the
uncertainties of the laboratorys measurement
standard(s) in the denominator [17]. A TUR
greater than 4:1 was thought to indicate a robust calibration process.
The TUR originated in the Navys Produc42

NCSLI Measure

The ratio of the span of the tolerance of a

measurement quantity subject to calibration,
to twice the 95% expanded uncertainty of the
measurement process used for calibration.

previous definitions. This difference centers

on the components of the denominator. In
Z540.3, the uncertainty in the denominator is
very specifically defined as the uncertainty
of the measurement process used in calibration. This definition has broader implications than historical definitions because it
includes elements of the UUT performance
(for example, resolution and process repeatability) in the denominator. Many laboratories have long assumed that the uncertainty
of the measurement process, as it relates to
the denominator of TUR, should encompass
all aspects of the laboratory standards, environmental effects, measurement processes,
etc., but not the aspects of the UUT. Historically, the TUR denominator reflected the
capability of the laboratory to make highly
accurate measurements, but this capability
was sometimes viewed in the abstract sense,
and was independent of any aspects of the
UUT. The redefined TUR in the Z540.3 includes everything that affects a laboratorys
ability to accurately perform a measurement
on a particular device in the expanded uncertainty, including UUT contributions. This
was reiterated to NASA in another response
from the NCSLI 174 Standards Writing Committee [19].
The new definition of TUR is meant to
serve as a single simplistic metric for evaluating the plausibility of a proposed compliance
test with regard to mitigating false accept
risk. No distinction is made as to where the
risk originates, it could originate with either
the UUT or the laboratory standard(s). A
low TUR does not necessarily imply that the
laboratory standards are not good enough.
It might indicate, however, that the measurement cannot be made without significant false
accept risk due to the limitations of the UUT
itself. Such might be the case if the accuracy
specification of a device is equal to its resolution or noise floor. This can prevent a reliable
pass/fail decision from being made.
When computing TUR with confidence
levels other than 95 %, laboratories have
sometimes attempted to convert the UUT
specifications to 2 before dividing by the
expanded uncertainty (2) of the measurement process. Or, equivalently, UUT specs
were converted to 1 for division by the
standard uncertainty (1) of the measurement process. Either way, this was believed
by some to provide a more useful apples-toapples ratio for the TUR. Efforts to develop
an equivalent or normalized TUR have been
www.ncsli.org

TECHNICAL PAPERS
documented by several authors [18, 20, 21,
22]. However, the integrity of a TUR depends
upon the level of effort and honesty demonstrated by the manufacturer when assigning
accuracy specifications to their equipment.
It is important to know if the specifications
are conservative and reliable, or if they were
produced by a marketing department that was
motivated by other factors.
6. Understanding Program Level False
Accept Risk

Figure 4. The probability density of possible measurement results.

Figure 5. Topographical contour map with tolerance limits (L) and regions
of incorrect compliance decisions.

43

NCSLI Measure

Investigating the dependency of false accept

risk on EOPR and TUR is well worth the effort involved. The reader is referred to several
papers that provide an excellent treatment of
the mathematics behind the risk requirements
at the program level [3, 4, 23, 24, 25]. These
publications and many others build upon the
seminal works on measurement decision risk
by Eagle, Grubbs, Coon, & Hayes [18, 26, 27]
and should be considered required reading.
This discussion is more conceptual in
nature, but a brief overview of some fundamental principles is useful. As stated earlier,
M&TE tolerance limits are often set by the
manufacturers accuracy specifications. The
device may be declared in-tolerance if the
UUT is observed to have a calibration result
eobs that is within the tolerance limits L. This
can be written as L eobs L. The observed
calibration result eobs is related to the actual or
true UUT error euut and the measurement process error estd by the equation eobs = euut + estd.
Note that the quantity euut is the parameter being sought when a calibration is performed,
but eobs is what is obtained from the measurement. The value of euut is always an estimate
due to the possibility of measurement process
errors estd described by uncertainty U95. It is
not possible to determine euut exactly.
Errors (such as euut and estd), as well as
measurement observations (such as eobs), are
quantities represented by random variables
and characterized by probability density
functions. These distributions represent the
relative likelihood of any specific error (euut
and estd) or measurement observation (eobs)
actually occurring. They are most often of
the Gaussian form or normal distribution and
are described by two parameters, a mean or
average , and a standard deviation . The
standard deviation is a measure of the variability or spread in the values from the mean.
The mean of all the possible error values
will be zero, which assumes systematic effects have been corrected. Real-world mea-

www.ncsli.org

TECHNICAL PAPERS
surements are a function of both () characterized by the UUT performance uut and the
measurement eobs with associated uncertainty
, where s obs = s uut + s std . The relative likelihood of all possible measurement results is
represented by the two dimensional surface
area created by the joint probability distribution given by (, eobs) = () (std). Figures 4 and 5 illustrate the concept of probability density of measurement and represent
the relative likelihood of possible measurement outcomes given the variables TUR and
EOPR. It is assumed that measurement uncertainty and the UUT distribution follow a
normal or Gaussian probability density function, yielding a bivariate normal distribution.
Figure 5 is a top-down perspective of Fig. 4,
when viewed from above.
The height, shape, and angle of the joint
probability distribution change as a function
of input variables TUR and EOPR. The dynamics of this are critical, as they define the
amount of risk for a given measurement scenario. The nine regions in Fig. 5 are defined
by two-sided symmetrical tolerance limits.
Risk is the probably of a measurement occurring in either the false accept regions or
the false reject regions. Computing an actual
numeric value for the probability (PFA or
PFR) involves integrating the joint probability density function over the appropriate two
dimensional surface areas (regions) defined
by the limits stated below. Incorrect (false)
acceptance decisions are made when euut >
|L| and L eobs L. In this case, the UUT
is truly out of tolerance, but is observed to
be in tolerance. Likewise incorrect (false) reject decisions are made when eobs >|L| and L
euut L, or where the UUT is observed to
be out of tolerance, but is truly in tolerance.
Integration over the entire joint probability
region will yield a value of 1, as would be
expected. This encompasses 100 % of the
volume under the surface of Fig. 4. When
the limits of integration are restricted to the
two false accept regions shown in Fig. 5, a
small portion of the total volume is computed
which represents the false accept risk as a
percentage of that total volume.
In the ideal case, if the measurement uncertainty was zero, the probability of measurement errors estd occurring would be zero.
The measurements would then perfectly reflect the behavior of the UUT and the distribution of possible measurement results would
be limited to the distribution of actual UUT
errors. That is, (obs) would equal (uut)
44

NCSLI Measure

and the graph in Fig. 5 would collapse to a

straight line at a 45 angle and the width in
Fig. 4 would collapse to a simple two dimensional surface with zero volume. However,
since real-world measurements are always
hindered by the probability of errors, observations do not perfectly reflect reality and
risk results. In this case, the angle is given by
s
tan(q ) = obs , where 45 90.
s uut

7. Efficient Risk Mitigation

In order for a calibration laboratory to comply

with Z540.3 (5.3b), the program level PFA
must not exceed 2 % and must be documented. However, computing an actual value for
PFA is not necessarily required when demonstrating compliance with the 2 % rule. To understand this, consider that the boundary conditions of PFA can be investigated by varying
the TUR and EOPR over a wide range of values and observing the resultant PFA. This is
best illustrated by a three dimensional surface
plot, where the x and y axis represent TUR
and EOPR, and the height of the surface on
the z-axis represents PFA (Fig. 6 and 7).
This surface plot combines both aspects
affecting false accept risk into one visual
representation that illustrates the relationship
between the variables TUR and EOPR. One
curious observation is that the program level
PFA can never be greater than 13.6 % for any
combination of TUR and EOPR. The maximum value of 13.6 % occurs when the TUR
is approximately 0.3:1 and the EOPR is 41
%. Any change, higher or lower, for either
the TUR or EOPR will result in a PFA lower
than 13.6 %.
One particularly useful observation is that,
for all values of EOPR, the PFA never exceeds 2 % when the TUR is above 4.6:1. In
Figures 6 and 7, the darkest blue region of the
PFA surface is always below 2 %. Even if the
TUR axis in the above graph were extended
to infinity, the darkest blue PFA region would
continue to fall below the 2 % threshold. Calibration laboratory managers will find this to
be an efficient risk mitigation technique for
compliance with Z540.3. The burden of collecting, analyzing, and managing EOPR data
can be eliminated when the TUR is greater
than 4.6:1.
This concept can be further illustrated by
rotating the perspective (viewing angle) of
the surface plot in Fig. 6, allowing the two
dimensional maximum outer-envelope or
boundary to be easily viewed. With this perspective, PFA can be plotted only as a func-

tion of TUR (Fig. 8). In this instance, the

worst-case EOPR is used whereby the maximum PFA is produced for each TUR.
The left-hand side of the graph in Fig.
8 might not appear intuitive at first. Why
would the PFA suddenly decrease as the
TUR drops below 0.3:1 and approaches zero?
While a full explanation is beyond the scope
of this paper, the answer lies in the number of
items rejected (falsely or otherwise) when an
extremely low TUR exists. This causes the
angle of the joint probability distribution to
rotate counter-clockwise away from the ideal
45 line, shifting areas of high density away
from the false accept regions illustrated in
Fig. 5. For a very low TUR, there are indeed
very few false accepts and very few correct
rejects. The outcome of virtually all measurement decisions is then distributed over
the correct accept and false reject regions as
approaches 90. It would be impractical for a
calibration laboratory to operate under these
conditions, although false-accepts would be
exceedingly rare.
Examining the boundary conditions of
the surface plot also reveals that the PFA is
always below 2 % where the true EOPR is
greater than 95 %. This is true even with extremely low TURs (even below 1:1). Again,
if the perspective of the PFA surface plot in
Fig. 6 is properly rotated, a two dimensional
outer-envelope is produced whereby PFA can
be plotted only as a function of EOPR (Fig.
9). The worst-case TUR is used for each and
every point of the Fig. 9 curve, maximizing
the PFA, and illustrating that knowledge of
the TUR is not required.
As was the case with a low TUR, a similar phenomenon is noted on the left-hand
side of the graph in Fig. 9; the maximum PFA
decreases for true EOPR values below 41 %.
As the EOPR approaches zero on the left
side, most of the UUT values lie far outside
of the tolerance limits. When the values are
not in close proximity to the tolerance limits,
the risk of falsely accepting an item is low.
Likewise on the right-hand side of the graph,
where the EOPR is very good (near 100 %),
the false accept risk is low. Both ends of the
graph represent areas of low PFA because
most of the UUT values have historically
been found to lie far away from the tolerance
limits. The PFA is highest, in the middle of
the graph, where EOPR is only moderately
poor, and where much of the data is near the
tolerance limits.

www.ncsli.org

TECHNICAL PAPERS

Figure 6. Surface plot of false accept risk as a function of TUR and EOPR.

Figure 7. Topographical contour map of false accept risk as a function of TUR and EOPR.
8. True Versus Observed EOPR

Until now, this discussion has been limited to

the concept of true EOPR. The idea of a
true EOPR implies that a value for reliability exists that has not been influenced by any
non-ideal factors, but of course, this is not the
case. In the calibration laboratory, reliability
data is collected from real-world observations or measurements. The measurements
45

NCSLI Measure

of UUTs are often made by comparing them

to reference standards with very low uncertainty under controlled conditions. But even
the best available standards have finite uncertainty, and the UUT itself often contributes
noise and other undesirable effects. Thus, the
observed EOPR is never a completely accurate representation of the true EOPR.
The difference between the observed and

true EOPR becomes larger as the measurement uncertainty increases and the TUR
drops. A low TUR can result in a significant
deviation between what is observed and what
is true regarding the reliability data [23, 28,
29, 30]. The reported or observed EOPR
from a calibration history includes all influences from the measurement process. In this
case, the standard deviation of the observed
www.ncsli.org

TECHNICAL PAPERS

Figure 8. Worst case false accept risk vs. TUR.

Figure 9. Worst case false accept risk vs. EOPR.

distribution is given by s obs = s uut 2 + s std 2

where and std are derived from statistically independent events. The corrected or
true standard deviation can be approximated by removing the effect of measurement
uncertainty and solving for s uut = s obs 2 - s std 2
where uut is the true distribution width represented by standard deviation.
The above equation shows that the stan46

NCSLI Measure

dard deviation of the observed EOPR data is

always worse (higher) than the true EOPR
data. That is, the reliability history maintained by a laboratory will always cause the
UUT data to appear to be further dispersed
than what is actually true. This results in an
89 % observed EOPR boundary condition
where the PFA is less than 2 % for all possible values of TUR6 (Fig. 10).

If measurement uncertainty is thought of

as noise, and the EOPR is the measurand,
then the observed data will have greater variability or scatter than the true value of the
6

When correcting EOPR under certain conditions,

low TUR values can result in imaginary values
for uut. This can occur where uut and std are
not statistically independent and/or the levels of
confidence associated with std and/or uut have
been misrepresented.

www.ncsli.org

TECHNICAL PAPERS

Figure 10. PFA assumes worst case TUR for true EOPR and observed EOPR.

EOPR. Measurement uncertainty always at face value (analogous to Type B or heuhinders the quest for accurate data; it never ristic evaluations). However, the influence of
helps. The true value of a single data point the measurement process is always present.
can be higher or lower than the measured This method of removing measurement unvalue, it is never known whether the mea- certainty from the EOPR data is a best estisurement uncertainty contributed a positive mate of the true reality or reliability which is
error or negative error. Therefore, it is not sought through measurement.
possible to remove the effect of measurement
uncertainty from a single measurement result. 9. Guardbanding
However, EOPR data is a historical collection It is sometimes helpful to establish acceptance
of many pass/fail compliance decisions that limits A at the time-of-test that are more strincan be represented by a normal probability gent than the manufacturers tolerance limits
distribution with a standard deviation obs. L. Acceptance limits are often called guardSometimes the measurement uncertainty std band limits or test-limits. It is only necessary
will contribute positive errors and sometimes to implement acceptance limits A, which differ from
the tolerance
limits L, when the false
it will contribute
negative
errors.
mean error.
easurement uncertainty
contributed
a positive
errorIforthe
negative
Therefore,
it
to remove the
effect eof
measurement uncertainty from a single
measurement
accept
risk is higher than desired or as part
of these
std errors is assumed to be zero, the
er, EOPR data is a historical collection of many pass/fail compliance
decisionsto keep risk below a specified
effect of measurement uncertainty on a popu- of a program
.
presented by a normal probability distribution with a standard deviation
level.
Acceptance
limits may be chosen to
of EOPR data
be removed
as previmeasurementlation
uncertainty
willcan
contribute
positive
errors and
sometimes
it
at either the bench level or the
ously
shown.
inverse normal
negative errors.
If the
mean The
of these
errors isfunction
assumedisto bemitigate
zero, therisk
effect
t uncertainty on
a population
of EOPR
data can
be removed
as previously
programshown.
level. PFA calculations may be used
used
to estimate
observed
EOPR
obs from
rmal function data
is used
to
estimate
from
observed
EOPR
data
[31]
to establish acceptance limits based on the
[31]
mandated risk requirements. In most instanc(2)
,
(2) bands are applied, the toleres, where guard
(
)
ance limits are temporarily tightened or
where
-1 represents
esents the inverse
normal
distribution.the inverse normal dis- reduced to create acceptance limits needed to
tribution.
meet a PFA goal. The subject of guardbands a numerical quantity
at by statistical
means
applied
data
EOPRarrived
is a numerical
quantity
arrived
at to empirical
ing is extensive and novel approaches exist
Type A evaluation
in
the
language
of
the
GUM
[32].
The
data
comes
from
by statistical means applied to empirical data for establishing
urements made over time rather than from accepting manufacturers
claims at face acceptance limits to mitigate
analogous
to a Type AHowever,
evaluation
the lan- of the
risk,
even where EOPR data is not available
us to Type B or
heuristic evaluations).
theininfluence
measurement
guage
of theofGUM
[32]. The
data comes
from [25].
ays present. This
method
removing
measurement
uncertainty
from the
EOPR in the simplified case of no
However,
stimate of the true
realitymeasurements
or reliability which
sought
through
repeated
madeisover
time
rath- measurement.
guardbanding, the acceptance limits A are set
er than from accepting manufacturers claims equal to the tolerance limits L (A = L ).
ing

helpful to establish acceptance limits A at the time-of-test that are more stringent
acturers tolerance
L. Acceptance
limits are often called guardband limits or
47 limits
| NCSLI
Measure
only necessary to implement acceptance limits A, which differ from the tolerance
the false accept risk is higher than desired or as part of a program to keep risk

One particularly useful method employing a guardbanding technique is described

in Method 6 of the Z540.3 Handbook [16,
25]. This method does not require EOPR
data to be available because it relies on using
worst-case EOPR, computed for a specified
TUR value. Using this approach, a guardband multiplier is computed as a function of
TUR. The acceptance limits are expressed
as follows: A = L MU95 , where A is the
newly established acceptance limits, L is the
original tolerance limits, U95 is the expanded
measurement process uncertainty, and M is
the multiplying factor that yields a risk of a
specified maximum target. Figure 11 graphs
guardband multipliers for varying levels of
risk. The risk level for Z540.3 is specified at
2 % but could vary depending upon the agreement with the customer. M2% was previously
calculated by Dobbert [25] by fitting a line
though the data points that mitigate risk to a
level of 2 % and is given by the following
simplified formula

M 2% = 1.04 - e0.38 ln(TUR)-0.54 . (3)

(3)

It can be seen that the line is a good fit for

the condition where 1 TUR 15. The intent
was to keep the equation simple while covering the range of TUR values that make physical sense. It has been shown in this paper that
for TUR 4.6, PFA is always < 2 %. To verify
that TUR = 4.6 is a boundary condition, set
M2% = 0 and solve for TUR. It is worth noting
www.ncsli.org

TECHNICAL PAPERS

Figure 11. Guardband multiplier for acceptable risk limits as a function of TUR.

that, for 4.6, the multiplier M2% is < 0. This

implies that a calibration lab could actually
increase the acceptance limits A beyond the
UUT tolerances L and still comply with the
2 % rule. While not a normal operating procedure for most calibration laboratories, setting
guard band limits outside the UUT tolerance
limits is possible while maintaining compliance with the program level risk requirement
of Z540.3. In fact, laboratory policies often
require items to be adjusted back to nominal
for observed errors greater than a specified
portion of their allowable tolerance limit L.
10. Conclusion and Summary

Organizations must determine if risk is to be

controlled for individual workpieces at the
bench level, or mitigated for the population of
items at the program level7. Computation of
PFA at the program level requires the integration of the joint probability density function.
The input variables to these formulas can be
reduced to EOPR and TUR. The 2 % PFA
maximum boundary condition, formed by either a 4.6:1 TUR or an 89 % observed EOPR,
can greatly reduce the effort required to manage false accept risk for a significant portion
of the M&TE submitted for calibration. Either or both boundary conditions can be leveraged depending on the available data, providing benefit to practically all laboratories.
However, there will still be instances where
the TUR is lower than 4.6: 1 and the observed
48

NCSLI Measure

EOPR is less than 89 %. In these instances,

it is still possible for the PFA to be less than
2 %. A full PFA computation is required to
show the 2 % requirement has not been exceeded. However, other techniques can be
employed to ensure that the PFA is held below
2 % without an actual computation.
There are six methods listed in the Z540.3
Handbook for complying with the 2 % false
accept risk requirement [16]. These methods
encompass both program level and bench
level risk techniques. This paper has specifically focused on some efficient approaches
for compliance with the 2 % rule, but it does
not negate the use of other methods nor imply
that the methods discussed here are necessarily the best. The basic strategies outlined here
for handling risk without rigorous computation of PFA are:

Analyze EOPR data. This will most likely be done at the instrument-level, as opposed to the test-point level, depending on
data collection methods. If the observed
EOPR data meets the required level of 89
%, then the 2 % PFA rule has been satisfied.
If this is not the case, then further
analysis is needed and the TUR must
be determined at each test point. If the
analysis reveals that the TUR is greater
than 4.6:1, no further action is necessary and the 2 % PFA rule has been met.

If neither the EOPR nor TUR threshold

is met, a Method #6 guardband can be
applied.

Compliance with the 2 % rule can be accomplished by either calculating PFA and/or
limiting its probability to less than 2% by the
methods presented above. If these methods
are not sufficient, alternative methods of mitigating PFA are available [16]. Of course, no
amount of effort on the part of the calibration
laboratory can force a UUT to comply with
unrealistic expectations of performance. In
some cases, contacting the manufacturer with
this evidence may result in the issuance of
revised specifications that are more realistic.
Assumptions, approximations, estimations, and uncertainty have always been part
of metrology, and no process can guarantee
that instruments will provide the desired accuracy, or function within their assigned tolerances during any particular application or
use. However, a well-managed calibration
process can provide confidence that an instrument will perform as expected and within
limits. This confidence can be quantified via
analysis of uncertainty, EOPR, and false accept risk. Reducing the number of assump7

Bayesian analysis can be performed to determine

the risk to an individual workpiece using both the
measured value on the bench and program-level
EOPR data to yield the most robust estimate of
false accept risk [31].

www.ncsli.org

TECHNICAL PAPERS

tions and improving the estimations involved during calibration can

not only increase confidence, but also reduce risk and improve quality.
11. Acknowledgements

The authors thank the many people who contributed to our understanding of the subject matter presented here. Specifically, the contributions of Perry King (Bionetics), Scott Mimbs (NASA), and Jim
Wachter (Millennium Engineering and Integration) at Kennedy Space
Center were invaluable. Several graphics were generated using PTCs
MathCad 14. Where numerical methods were more appropriate,
Microsoft Excel was used incorporating VBA functions developed
by Dr. Dennis Jackson of the Naval Surface Warfare Center in Corona,
California.
12. References

[1]

JCGM, International vocabulary of metrology Basic

and general concepts and associated terms (VIM), JCGM
200:2008, 2008.
ANSI/NCSLI, Requirements for the Calibration of Measuring
and Test Equipment, ANSI/NCSL Z540.3:2006, 2006.
D. Deaver and J. Somppi, A Study of and Recommendations
for Applying the False Acceptance Risk Specification of
Z540.3, Proceedings of the Measurement Science Conference,
Anaheim, California, 2010.
H. Castrup, Risk Analysis Methods for Complying with
Z540.3, Proceedings of the NCSL International Workshop and
Symposium, St. Paul, Minnesota, 2007.
M. Dobbert, A Pragmatic Method for Pass/Fail Conformance
Reporting that Complies with ANSI Z540.3, ISO 17025, and
ILAC-G8, Proceedings of the NCSL International Workshop
and Symposium, Providence, Rhode Island, 2009.
ANSI/NCSLI, Calibration & Measurement & Test Equipment
- General Requirements, ANSI/NCSL Z540.1: 2002, 2002
ISO/IEC, General requirements for the competence of testing
and calibration laboratories, ISO/IEC 17025:2005(E), 2005.
A2LA, Specific Requirements: Calibration Laboratory
Accreditation Program, A2LA:R205, 2011.
ILAC, Guidelines on Assesment and Reporting of Compliance
with Specification (based on measurements and tests in a
laboratory), ILAC-G8:1996, 1996.
UKAS, The Expression of Uncertainty and Confidence in
Measurement (Appendix M), UKAS:M3003, 2007.
ASME, Guidelines for Decision Rules: Considering
Measurement Uncertainty in Determining Conformance to
Specifications, ASME B89.7.3.1-2001, 2001.
ISO, Geometrical Product Specifications (GPS) - Inspection by
measurement of workpieces and measuring equipment - Part 1:
Decision Rules for proving conformance or non-conformance
with specifications, ISO-14253-1:1998(E), 1998.
ASME, Measurement Uncertainty Conformance Testing: Risk
Analysis, ASME B89.7.4.1-2005 (Technical Report), 2006.

[2]
[3]

[4]

[5]

[6]
[7]
[8]
[9]

[10]
[11]

[12]

[13]

49

NCSLI Measure

[14] NCSLI, Response to NASA Interpretation Request (IR2),

NCSLI 174 Standards Writing Committee, March 2008.
[15] S. Mimbs, Request for Interpretation; Correspondence Letter,
July 2007.
[16] ANSI/NCSLI, Handbook for the Application of ANSI/NCSL
Z540.3:2006 - Requirements for the Calibration of Measuring
and Test Equipment, ANSI/NCSL Z540.3 Handbook, 2009.
[17] J. Bucher, ed., The Metrology Handbook, American Society for
Quality, Measurement Quality Division, ASQ Quality Press,
ISBN 0-87389-620-3, 2004.
[18] J. Hayes, Factors Affecting Measuring Reliability, U.S.
Naval Ordnance Laboratory Technical Memorandum No. 63106, October 1955.
[19] NCSLI, Response to NASA Interpretation Request (IR1),
NCSLI174 Standards Writing Committee, March 2008.
[20] M. Nicholas and L. Anderson, Guardbanding Using Automated
Calibration Software, Proceedings of the NCSL International
Workshop and Symposium, Salt Lake City, Utah, 2004.
[21] Fluke Corporation, Calibration: Philosopy in Practice, ISBN:
978-0963865007, May 1994.
[22] T. Skwircznski, Uncertainty of the calibrating instrument,
confidence in the measurement process and the relation between
them, International Organization of Legal Metrology (OIML)
Bulletin, vol. XLII, no.3, July 2001.
[23] NASA, Estimation and Evaluation of Measurement Decision
Risk, NASA Measurement Quality Assurance Handbook
ANNEX 4, NASA-HDBK-8739.19-4, July 2010.
[24] M. Dobbert, Understanding Measurement Decision Risk,
Proceedings of the NCSL International Workshop and
Symposium, St. Paul, Minnesota, 2007.
[25] M. Dobbert, A Guard Band Strategy for Managing False
Accept Risk, Proceedings of the NCSL International Workshop
and Symposium, Orlando, Florida, 2008, .
[26] A. Eagle, A Method for Handling Error in Testing and
Measuring, Industrial Quality Control, March 1954.
[27] F. Grubbs and H. Coon, On Setting Test Limits Relative to
Specification Limits, Industrial Quality Control, March 1954.
[28] S. Mimbs, Measurement Decision Risk - The Importance
of Definitions, Proceedings of the Measurement Science
Conference, Anaheim, California, 2008.
[29] J. Ferling, The Role of Accuracy Ratios in Test and
Measurement Processes, Proceedings of the Measurement
Science Conference, Long Beach, California, 1984.
[30] I. Lira, A Bayesian approach to the consumers and producerss
risk in measurement, Metrologia, vol. 36, pp. 397-402,
October 1999.
[31] H. Castrup, Analytical Metrology SPC Methods for ATE
Implementation, Proceedings of the NCSL Workshop and
Symposium, Albuquerque, New Mexico, 1991.
[32] ISO/IEC, Uncertainty of measurement -- Part 3: Guide to the
expression of uncertainty in measurement, ISO/IEC Guide 983:2008, 2008.

www.ncsli.org