I'm testing a new instrument with the traditional instrument and taking measurements on the same subject using both instruments. I want to be able to make a statement that "at least 95% of the differences between the two is no more than 0.05 units".

To answer this, I did some searching, and it seems that a tolerance interval would be appropriate, but I'm having trouble creating a decision rule (one-sided) to test this statement. Any thoughts? Thanks!


@StéphaneLaurent, after looking at your post, it seems very much like what I'd like to accomplish - both measurements have some sort of random noise associated with them, and I'm interested in the differences.

For the both of you, let me see if I understand the concept of the TI correctly: If I observe an upper tolerance bound ($p$ = 0.95, $1-\alpha$ = 0.95), say U, and the value of U is less than the allowable threshold (0.05 units), then I can say with 95% confidence that at least 95% of the measurements fall within the threshold...Is this the correct understanding and can I use this as my decision rule?

I want to make a decision rule to see whether or not I should keep going with my instruments or if I should do additional testing. Thanks!

  • $\begingroup$ Your statement must be conditional on the set of samples that will be submitted to the instruments. What are you assuming about that set? $\endgroup$
    – whuber
    Commented Jul 4, 2013 at 20:18
  • $\begingroup$ Do you take measurements always around the same value or rather in a possible range of values ? In the second case you might use a Deming model and see stats.stackexchange.com/questions/44219/… for tolerance intervals. $\endgroup$ Commented Jul 4, 2013 at 20:31
  • $\begingroup$ @whuber, I'm not quite sure what you mean - I have 30 observations for each of the instruments. I'm assuming that the instruments operate with random Gaussian error $\endgroup$ Commented Jul 5, 2013 at 17:51
  • $\begingroup$ You say you want to make a statement about "differences between the two" instruments. We can expect the sizes of those differences to depend on the actual values of the samples submitted for measurement. Therefore, the proportion of differences that are no more than 0.05 units (for example) will depend on the actual distribution of the values of the samples. Until you specify what that distribution will be, it will be difficult to give precise answers to your question. $\endgroup$
    – whuber
    Commented Jul 5, 2013 at 17:55
  • $\begingroup$ I assume that those differences will be normally distributed. Also please see edit above. Thanks! $\endgroup$ Commented Jul 5, 2013 at 18:02

1 Answer 1


Let's assume that you have no evidence that the size (or sign) of the difference varies with the measurement value. (Such variation is typical in instruments, where the often the measurement error is proportional to the underlying value, so this is not just a theoretical concern.) When this is the case, you may view the observed differences (as computed from your data) as a set of independent results from some underlying distribution $F$ of differences between the first instrument and the second.

A "tolerance limit" is actually a confidence limit for a specified quantile of $F$, often written as the $1-\gamma$ quantile $F_{1-\gamma}$ when you're interested in an "upper" TL: $\gamma$ is typically small, such as 5% or 1%. All confidence limits are numbers computed from the data; if we write the data as a vector $X$, we might symbolically write the confidence limit as $t(X)$. It's usually given as some kind of formula to be applied to whatever values $X$ actually has.

As with any confidence limit/interval procedure, there are four things you could demand with a given "confidence" of $1-\alpha$ (where, once again, $\alpha$ tends to be small in practice):

  1. An upper confidence limit for $F_{1-\gamma}$. This is constructed to that prior to making measurements, the limit has a $1-\alpha$ chance of exceeding the true quantile:

    $${\Pr}_F(t(X) \ge F_{1-\gamma}) \ge 1-\alpha.$$

    This statement must hold no matter what $F$ might happen to be, but it must also be the case that for any $\alpha' \lt \alpha$, there must be at least one $F$ for which the chance actually is less than $1-\alpha'$: this forces $t(X)$ to tend to be as small as possible while still meeting this criterion.

    The interpretation is that once you compute that limit $t(X)$, you can say

    I used a procedure that had a $1-\alpha$ chance of exceeding the $1-\gamma$ percentile of the underlying distribution. Because $\alpha$ is small, we can be confident that $t(X)$ is larger than at least $1-\gamma$ of the underlying values. This means that over a very long period of use of the first of these instruments, we can expect it not to exceed what the other would have read by more than $t(X)$ in at least $100(1-\gamma)$% of its uses.

    It's a mouthful, but I hope that last sentence is sufficiently clear.

  2. A lower confidence limit for $F_{1-\gamma}$. This is constructed similarly to the upper one, mutatis mutandis, and is intended to be fairly certain of underestimating $F_{1-\gamma}$; it's probably not applicable here.

  3. A two-sided confidence interval. This is usually designed to be symmetric in the sense that the chance of exceeding the upper limit of the interval equals the chance of lying below the lower limit of the interval.

  4. An estimate. This is your "best" (in some sense) estimate of $F_{1-\gamma}$. If it's any good, then for all or most of the possible $F$, there will be roughly equal chances of overestimating and underestimating the quantile. It's possible that's what is desired in this case.

Because negative and positive differences are of equal concern (presumably), what appears to be needed here actually is a (two-sided) tolerance interval for the differences. This is a pair of statistics $l(X)$ and $u(X)$ forming the interval $(l(X), u(X)]$. Its defining criterion is

$${\Pr}_F(F(u(X)) - F(l(X)) \ge 1-\gamma) \ge 1-\alpha.$$

In words: the interval is constructed so it has a chance of at least $1-\alpha$ of covering $100(1-\gamma)$% of the underlying distribution $F$, regardless of what that distribution might actually be.

Because confidence limits for $F_{1-\gamma/2}$ and $F_{\gamma/2}$ are typically correlated, you can't really play the same game with the coverage that is played with confidence limits by combining an upper $1-\gamma/2$ tolerance limit with a lower $\gamma/2$ tolerance limit to make up a $1-\gamma$ tolerance interval. But, for lack of simple formulas, that's what many people do anyway as an approximation.

The interpretation of this is

over a very long period of use of the first of these instruments, we can expect that the reading on the second instrument (had it actually been obtained) would have been between $l(x)$ greater and $u(x)$ less in at least $100(1-\gamma)$% of its uses.

How to compute $t(X)$ or $(l(X), u(X))$ depends on the possibilities for $F$. There are various (and different procedures) depending on whether $F$ is known to be Normal (but of unknown mean and variance), or Gamma, or whatever. There are other procedures for the case when few (if any) assumptions are made about $F$ ("non-parametric tolerance limits"). The trade-off between the assumptions you're willing to make and the amount of data you need is particularly stark. The parametric procedures tend to be sensitive to the parametric assumptions (because with small $\gamma$ you're reaching pretty far into the tails of the distribution). The non-parametric procedures are robust but require a lot of data and still are not very resistant to outlying values. E.g., to get a reasonable 95% confidence, 95% coverage two-sided tolerance interval--even one that is resistant to just a single outlier--typically requires hundreds of observations.

For a discussion of tolerance limits and intervals and a guide to an R package for computing them see Derek S. Young, tolerance: an R package for estimating tolerance intervals, J. Stat. Soft. August 2010 vol 36 issue 5.

  • 1
    $\begingroup$ Wow thanks for the detailed answer. The interpretations were particularly very helpful and got me to thinking as to whether the TI is really appropriate for this study. It might seem like a simpler approach would be sufficient. Again, thanks for the time you put into this! I really appreciate it. $\endgroup$ Commented Jul 5, 2013 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.