Here's a hypothetical situation a colleague and I were discussing, and I'd be interested to hear your opinion(s) on this approach to utilizing confidence intervals:
We are discussing a method of putting a "confidence level" on test failure predictions based on the confidence intervals computed for the mean and standard deviations of samples taken during production.
A "good" sample (sufficient for the half-width desired from a mostly stable process) is taken, and the sample mean & sample standard deviation computed. These are performed weekly at least, daily sometimes depending on volume needed through production each week. A variety of factors can influence the results, most are contained but there is still a some noise (ongoing work is always seeking to reduce it by experimentation to eliminate newly-discovered special causes) so I would categorize the overall process as mostly stable (long-term shifts only). Initially, 98% confidence intervals for both mean and standard deviation are computed as well during the sampling. Here begins the discussion:
Since a confidence interval expresses a probability that repeated executions of that sample selection would either encompass the true population parameter or not (i.e. the ratio over many, many samples for which the computed CI does encompass the parameter vs. doesn't encompass it), a 98% CI represents a 2% risk that any sample you take will not actually encompass the true parameter (µ or σ).
Q1) For mound-shaped data (normal or very near, AD normality test p-value of no worse than .05 let's say), is that CI risk equally distributed on both sides of the CI interval, i.e. a 2% risk is approximately equally distributed as marginal risks of (a) 1% chance of the true parameter being below your computed CI and (b) 1% chance of the true parameter being above your computed CI? I would think it would be since the CI is a two-tailed interval estimate, but I've never analyzed it.
Continuing the discussion:
For considering the case that both the CI of the mean and the CI of the standard deviation are wrong, and both are wrong on the low side, so (b) of Q1 above for both statistics (call this b1 and b2), leads to Q2 and Q3:
Q2) Would the chance of that event occurring on a future sample, i.e. the marginal probability of having underestimated µ via x-bar and simultaneously having underestimated σ with s, be the compound probability p(b1)*p(b2)?
Q3) Are those exclusive events? The estimations are possibly not independent, since s is computed via x-bar, so is that treatment of these probabilities of occurrence a violation of the assumptions of independence required for compound probability calculation?
Finishing the discussion:
Q4) Assuming that the probabilities of underestimating both parameters simultaneously can be treated as independent events, then does it follow that the singular risk of underestimating one parameter via a 98% confidence interval (a 1% chance assuming from Q1 that the chance is equally divided on both sides of normal-shaped data, (1-.98)/2 = 0.01) is the same as the compound risk of underestimating both parameters via an 80% confidence interval, as[(1-0.8)/2]*[(1-0.8)/2] = 0.01?