Here's a hypothetical situation a colleague and I were discussing, and I'd be interested to hear your opinion(s) on this approach to utilizing confidence intervals:

We are discussing a method of putting a "confidence level" on test failure predictions based on the confidence intervals computed for the mean and standard deviations of samples taken during production.

A "good" sample (sufficient for the half-width desired from a mostly stable process) is taken, and the sample mean & sample standard deviation computed. These are performed weekly at least, daily sometimes depending on volume needed through production each week. A variety of factors can influence the results, most are contained but there is still a some noise (ongoing work is always seeking to reduce it by experimentation to eliminate newly-discovered special causes) so I would categorize the overall process as mostly stable (long-term shifts only). Initially, 98% confidence intervals for both mean and standard deviation are computed as well during the sampling. Here begins the discussion:

Since a confidence interval expresses a probability that repeated executions of that sample selection would either encompass the true population parameter or not (i.e. the ratio over many, many samples for which the computed CI does encompass the parameter vs. doesn't encompass it), a 98% CI represents a 2% risk that any sample you take will not actually encompass the true parameter (µ or σ).

Q1) For mound-shaped data (normal or very near, AD normality test p-value of no worse than .05 let's say), is that CI risk equally distributed on both sides of the CI interval, i.e. a 2% risk is approximately equally distributed as marginal risks of (a) 1% chance of the true parameter being below your computed CI and (b) 1% chance of the true parameter being above your computed CI? I would think it would be since the CI is a two-tailed interval estimate, but I've never analyzed it.

Continuing the discussion:

For considering the case that both the CI of the mean and the CI of the standard deviation are wrong, and both are wrong on the low side, so (b) of Q1 above for both statistics (call this b1 and b2), leads to Q2 and Q3:

Q2) Would the chance of that event occurring on a future sample, i.e. the marginal probability of having underestimated µ via x-bar and simultaneously having underestimated σ with s, be the compound probability p(b1)*p(b2)?

Q3) Are those exclusive events? The estimations are possibly not independent, since s is computed via x-bar, so is that treatment of these probabilities of occurrence a violation of the assumptions of independence required for compound probability calculation?

Finishing the discussion:

Q4) Assuming that the probabilities of underestimating both parameters simultaneously can be treated as independent events, then does it follow that the singular risk of underestimating one parameter via a 98% confidence interval (a 1% chance assuming from Q1 that the chance is equally divided on both sides of normal-shaped data, (1-.98)/2 = 0.01) is the same as the compound risk of underestimating both parameters via an 80% confidence interval, as[(1-0.8)/2]*[(1-0.8)/2] = 0.01?



Some quick thoughts, though I hope someone else comes by with a more thorough answer later.

Q1) "Is the chance of over and under estimating equal?"

This depends entirely on how the interval is computed. There are (infinitely) many intervals which contain $x$ with probability 0.98, and only one of them is 'evenly distributed'. I expect that symmetry of the distribution is a more important property than 'mound shaped', and that you won't care about the amount of asymmetry if the tails are exponentially small.

Q2) "Does p(b1,b2) = p(b1)p(b2)?"

Clearly this is true if you estimate $\hat{\mu}$ and $\hat{\sigma}$ from independent samples of the data. In which case, the events $b1$ and $b2$ will be independent and $p(b1,b2) = p(b1)p(b2)$. This may not be the case, and I don't think that product form will hold in general if $\hat{\mu}$ and $\hat{\sigma}$ are not independent.

Q3) " Are b1 and b2 exclusive? "

No they are not exclusive. Suppose that I sample the same values of $x = \mu + \epsilon$ ten times. Then $\hat{\mu} = \mu+\epsilon > \mu$, but $\hat{\sigma} = 0 < \sigma$. Any particular example will happen with probability zero, but there is a non-zero probability that you overestimate $\mu$ and underestimate $\sigma$ if you use the same data (and vice versa).

Q4) "Is the chance of underestimating one parameter using a 0.98 interval the same as underestimating two parameters using a 0.8 interval?"

No - your calculations need to include the facts that (a) there is only a 99% chance of not underestimating the other parameter and (b) there are two ways you can underestimate one parameter. Namely, for 98% CI the probability of underestimating only one parameter is 0.01*0.99 + 0.99*0.01 = 0.0198, and the chance of underestimating two parameters using an 80% interval is 0.1*0.1 = 0.01.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.