6
$\begingroup$

I would like to know to which degree the width of a confidence interval can be used as a measure of variability of a statistic.

For instance, the confidence interval for the mean is given by $[\bar{x}-t^*\frac{s}{\sqrt{n}},\bar{x}+t^*\frac{s}{\sqrt{n}}]$ and the width of the confidence interval can be calculated accordingly:

$$\bar{x}+t^*\frac{s}{\sqrt{n}}-(\bar{x}-t^*\frac{s}{\sqrt{n}})=2t^*\frac{s}{\sqrt{n}}$$

For a 95% confidence interval, about the quarter (actually $\frac{1}{3.91993}$) of the width corresponds to the standard deviation of the sample mean due to the 68–95–99.7 rule. My experiments have shown that, compared to the point estimate of the standard deviation of the sample mean $\frac{s}{\sqrt{n}}$, the width of the confidence interval tends to overestimate the variability, which would be a desirable property for my application.

In the following log plot, you can see how the different statistics behave for an increasing number of normally distributed samples. The blue line represents the population standard deviation $\frac{\sigma}{\sqrt{n}}$, the orange line represents the point estimate $\frac{s}{\sqrt{n}}$, and the green line represents the scaled confidence interval width $\frac{2}{3.91993}t^*\frac{s}{\sqrt{n}}$.

Log plot of the population standard deviation, the point estimate of the standard deviation, and the scaled confidence interval width.

I am primarily interested in estimating the variability of the sample variance, especially for arbitrary (non-normal) distributions. There are several papers that deal with the specific problem of creating confidence intervals for such distributions (e.g., here).

In my experiments, the behaviors of the confidence interval for the variance and the confidence interval for the mean were comparable: both tended to overestimate the variability (compared to the respective point estimate). Therefore, I decided to use the confidence interval for the mean as example, as it is more demonstrative.

I am no statistician and I have no idea whether using the confidence interval width instead of the point estimate of the standard deviation is a "legitimate" thing to do. My intuitive explanation is that when using the confidence interval width, information about uncertainty at lower sample sizes is factored in. For the confidence interval for the mean, it is done through the $t^*$ value.

I would appreciate it if someone can elaborate on this and provide more than an intuitive explanation. Especially with regards to statistics that have a non-normal distribution (e.g., the variance), where the correspondence between confidence interval width and standard deviation is not that straightforward as for the normal distribution.

The mathematical notation I have used for this question is according to convention. Here is an overview.

$\endgroup$
  • 1
    $\begingroup$ I've upvoted your question as I'd like to see a detailed answer to it as well. However, I would like to note that confidence intervals are not always calculated the same way, and in some situations, some ways might even be considered better than others. Consider the Agresti-Coull adjusted Wald confidence intervals (compared to the regular Wald CIs) for proportions, or likelihood profile CIs compared to regular confidence intervals from regression models. As such, it is the context and goal of the (desired) statistic which are required to decide on 'suitability'. Could you elaborate on this? $\endgroup$ – IWS Oct 2 '17 at 14:58
  • $\begingroup$ I've edited my question. I hope it makes things clearer in this regard. $\endgroup$ – Hiro Oct 2 '17 at 15:58
1
$\begingroup$

In brief, it's not a bad measure, but there are some things to be aware of.

For (almost) any given data generating process, increasing the number of data will decrease the width of the confidence interval. Under some extra assumptions (those necessary for the Central Limit Theorem), the width will, asymptotically, follow an inverse square-root law. Even if the underlying data are normal, the maximum likelihood estimator of the variance and the unbiased estimator are not the same (which is the source of the systematic bias you show in your plot). This is due to the denominators being $n$ and $n-1$ respectively, so if you use the MLE you are out by a factor of $n/n-1$.

Even though the distribution of the estimator of the variance is not normal, it does asymptotically approach normality and still satisfies the Central Limit Theorem as long as your data satisfy $E(x^2)<\infty$ (i.e. the data actually has a variance so the estimator has something to estimate).

$\endgroup$
  • $\begingroup$ The $s$ I've used is the square root of the unbiased estimate of the variance and the bias (i.e., the difference between the orange and the green line) in the plot is solely due to the factor $\frac{2}{3.91993}t^*$ which is used to construct the confidence interval. Therefore, I'm not sure to which factor $\frac{n}{n-1}$ you are referring to. Can you be more specific in this regard? $\endgroup$ – Hiro Oct 2 '17 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.