In a first course to statistics, confidence interval calculation using the Central Limit Theorem is introduced.
- Are confidence intervals sensible in practice? That is, given a 99% confidence interval $[a,b]$ for the true mean $\mu$ of a quantity $Q$ using a sample of size $n$, and assuming that data collection was done with enough care, has there been a good track record* for whether one can reliably say that $\mu \in [a,b]$?
- After accounting for the error from Central Limit Theorem**, can the probability that $\mu \in [a,b]$ be calculated in practice? One way this can happen is that $\mu \in [a,b]$ holds with probability at least $1-\delta$, assuming that one samples at least $n \ge N_\delta$ points. Then is such an estimate for $N_\delta$ of any practical significance, or is it much larger than the value of $n$ typically required for a decent inference in practice?
*The questions above are somewhat vague, but I think they are important questions. I'd be pleased to hear your thoughts, even if it is only a specific interpretation of the above. Restricting the above questions of "reliability" to specific types of inferences is also welcome (e.g. medical diagnosis, party affiliation, presidential rating, energy of a particle, ...)
**One version of the Berry-Essen theorem quantifies such an error. Suppose $X_1, \cdots X_n$ are i.i.d. samples drawn from a distribution with the first three moments finite and equal to $0, \sigma^2, \rho$. Let $F_n(x)$ be the CDF of the rescaled sample mean $\frac{\sqrt{n}}{\sigma} \cdot \frac{X_1 + \cdots + X_n}{n}$, and $\Phi(x)$ be the CDF of the standard normal. Then there is a constant $C > 0$ such that (By Shevtsova 2011, one may use $C = 0.5$), for any $x$ and $n$, $$| F_n(x) - \Phi(x) | < \frac{C \rho}{\sigma^3 \sqrt{n}} $$ One can use this to bound the error in estimating tail sums with $\Phi$. But we need to estimate $\sigma$ and $\rho$, which involves another statistical inference. Being mostly new to statistics, I personally don't know if reliably estimating $\sigma$ and $\rho$ can be done in practice. Is an approach such as this meaningful at all in practice?