# Tolerance interval and significant probability

Let's take for simplicity an iid sample from a Gaussian random variable $Y \sim \mathcal{N}(\mu, \sigma^2)$. Let $y_0 \in \mathbb{R}$. The $(1-\alpha, p)$-bound of the lower one-sided (i.e. right-sided) tolerance interval is a $(1-\alpha)$-lower confidence bound of the quantile $q_p$ of the normal distribution $\mathcal{N}(\mu, \sigma^2)$. Therefore, if this tolerance bound is higher than $y_0$, one can say that the exceedance probability $\Pr(Y > y_0)$ is $\alpha$-significantly higher than $p$.

Now take two real numbers $y_1 < y_2$. If the two-sided $(1-\alpha,p)$-tolerance interval falls inside the interval $[y_1, y_2]$, does it make sense to say that the probability $\Pr\bigl(Y \in [y_1, y_2]\bigr)$ is significantly higher than $p$ at the $\alpha$ level?

• What happens as $y_1$ approaches arbitrarily close to $y_2$? The probability in question ought to grow vanishingly small. Any procedure that concludes this probability is significantly greater than any given positive $p$ would be awfully suspect!
– whuber
Aug 29, 2017 at 14:59
• @whuber I'm not comfortable with this question (that's why I ask it ^^) but I don't follow your reasoning. If the interval $[y_1,y_2]$ is small, and if the tolerance interval falls inside it, that means that the (estimate of the) standard deviation is small, and then the probability is not small. Aug 30, 2017 at 8:35
• I don't follow your conclusion, and perhaps that amounts to not understanding what you mean by "the two-sided tolerance interval." Usually that is understood to be a confidence interval for the quantile $q_p$. Thus, you cannot draw any conclusion about the SD of $Y$: like most confidence intervals, that one shrinks down to zero as the amount of data grows, independently of whatever value the SD of $Y$ might be (even if the SD is infinite!).
– whuber
Aug 30, 2017 at 12:21
• A lower (upper) tolerance limit is a lower (upper) confidence bound about the quantile. But this is not true for the two-sided tolerance interval (usually called the "tolerance interval", shortly). Aug 30, 2017 at 12:32
• Why don't you follow my conclusion ? The length of the tolerance interval is $2k\hat\sigma$. So I simply say that $\hat\sigma$ is small if this interval is contained in a small interval. Aug 30, 2017 at 12:34

No this is not true.

# One-sided case

Therefore, if this tolerance bound is higher than $y_0$, one can say that the exceedance probability $\Pr(Y > y_0)$ is $\alpha$-significantly higher than $p$.

Let's check this claim with the help of simulations. We simulate $Y_i \sim_{\text{iid}} \mathcal{N}(0,1)$, we take $y_0=1$, and we take the lower tolerance bound with coverage $p=\Pr(Y_i>y_0)$. Then the probability that the lower tolerance bound is higher than $y_0$ should be $\alpha$.

y0 <- 1
p <- 1-pnorm(y0)
nsims <- 200000
n <- 10
k <- tolerance::K.factor(n, alpha=0.05, P=p, side=1)
test <- logical(nsims)
for(i in 1:nsims){
Y <- rnorm(n)
bound <- mean(Y) - k*sd(Y)
test[i] <- bound > y0
}


This is confirmed by the simulations:

> mean(test)
[1] 0.049805


# Two-sided case

Now take two real numbers $y_1 < y_2$. If the two-sided $(1-\alpha,p)$-tolerance interval falls inside the interval $[y_1,y_2]$, does it make sense to say that the probability $\Pr\bigl(Y \in [y_1, y_2]\bigr)$ is significantly higher than $p$ at the $\alpha$ level?

Let's check with simulations that this is not true. As for the one-sided case, if we take $p = \Pr\bigl(Y_i \in [y_1, y_2]\bigr)$, then the probability that the two-sided $(1-\alpha,p)$-tolerance interval falls inside the interval $[y_1,y_2]$ should be $\alpha$ if the claim were true.

y1 <- -1; y2 <- 2
p <- pnorm(y2)-pnorm(y1)
nsims <- 200000
n <- 10
k <- tolerance::K.factor(n, alpha=0.05, P=p, side=2, method="EXACT")
test <- logical(nsims)
for(i in 1:nsims){
Y <- rnorm(n)
bounds <- mean(Y) + k*c(-1,1)*sd(Y)
test[i] <- bounds[1] > y1 && bounds[2] < y2
}


This is not confirmed:

> mean(test)
[1] 0.01193