0
$\begingroup$

In my most recent statistics class, the general1 frequentist approach to constructing confidence intervals2 for an estimator (of any kind) was described as follows:

  1. Specify the estimator and the data generating process (i.e., the likelihood $\mathcal{L}(x; \theta)$).
  2. Apply the estimator to observed data to obtain the estimate.
  3. Calculate the distribution of the estimator, assuming the estimate obtained in Step 2 is the true value of the estimand, and output the appropriate cutoffs to give a (central) $(1 - \alpha)$–confidence-interval.

For instance, in the case of MLE estimators, one would proceed as follows:

  1. Construct the likelihood function $\mathcal{L}(x; \theta)$ for the data generating process.
  2. Given observed data $X$, find the maximum likelihood estimate $\theta_{\text{MLE}} = \arg\min_{\theta} \mathcal{L}(X; \theta)$.
  3. Calculate the distribution of the estimator $\min_{\theta} \mathcal{L}(\tilde X; \theta)$ assuming $\tilde X$ has the distribution implied by $\theta_{\text{MLE}}$—e.g., the density of $\tilde X$ is given by $\mathcal{L}(x; \theta_{\text{MLE}})$—and then find $b_{\text{lower}}$ and $b_{\text{upper}}$ such that $\Pr(\arg\min_{\theta} \mathcal{L}(\tilde X; \theta) < b_{\text{lower}}) = \Pr(\arg\min_{\theta} \mathcal{L}(\tilde X; \theta) > b_{\text{upper}}) = \tfrac \alpha 2$ and output the interval $[b_{\text{lower}}, \, b_{\text{upper}}]$.

This process makes a lot of sense to me as a "theory" of frequentist CIs, but the most basic form of frequentist CI—viz., a CI for a sample mean—doesn't fit with this procedure.

In particular, if we assume that $X = (X_1, \ldots, X_n)$ represents $n$ i.i.d. draws from a normal $\mathcal{N}(\mu, \sigma^2)$, then we have that \begin{align} \mu_{\text{MLE}} &= \tfrac 1 n \sum_{i = 1}^n X_i \\ \sigma_{\text{MLE}} &= \sqrt{\tfrac 1 n \sum_{i = 1}^n (X_i - \mu_{\text{MLE}})^2} \end{align} Therefore, Step 3 suggests that we should look at the distribution of $\sum_{i = 1}^n Y_i$ where $Y_i \sim \mathcal{N}(\mu_{\text{MLE}}, \sigma_{\text{MLE}}^2)$; the distribution of this sample mean is, of course, $\mathcal{N}(\mu_{\text{MLE}}, \tfrac {\sigma_{\text{MLE}}^2} {n})$. However, the standard advice is to construct the confidence interval as if the sample mean were drawn from the $t$-distribution with $n-1$ degrees of freedom (appropriately scaled and recentered). This gives a slightly wider confidence interval which (based on a bunch of simulation I did and a century of received statistical wisdom) generally comes closer to nominal coverage.

Is there any way to make sense of this standard advice in light of the "general theory of frequentist confidence intervals" laid out above? Is this "general theory of frequentist confidence intervals" not actually the general theory at all?


1 I realize that this is not actually the most general formulation mathematically. I just mean that it was the "most general" in the sense of, "This is how we think about constructing confidence intervals in a frequentist setting."

2 "Frequentist confidence intervals" is, perhaps, tautological if you distinguish between "confidence intervals" and "credible intervals"—I just mean error bars that are not Bayesian (or Fiducial or ...).

$\endgroup$
12
  • 1
    $\begingroup$ You are overlooking the fact that $\sigma^2_{MLE}$ is an estimate too; if it were actually known, you would be correct, and the Normal distribution as you suggest would be appropriate. The distribution of $(\mu_{MLE}-\mu)/\sigma_{MLE}$ on the other hand is $t_{n-1}$. $\endgroup$
    – jbowman
    Oct 19, 2021 at 13:46
  • $\begingroup$ I think I'm maybe still missing something. Assuming, for the moment, that the data are drawn from a normal distribution, the sample mean and sample variance are independent. Therefore, when we get to Step 3, marginalizing over $\sigma_{\text{MLE}}^2$, we're still left with a normal distribution for $\mu_{\text{MLE}}$. So it still seems like the standard advice for constructing CIs for sample means and the standard story for constructing frequentist CIs laid out above aren't compatible. $\endgroup$
    – jgaeb
    Oct 19, 2021 at 17:07
  • $\begingroup$ You haven't marginalized over $\sigma^2_{MLE}$ in step 3. Your MLE is two-dimensional, and the $t$ statistic is based on both elements of it. You also seem unclear about what a likelihood function is - it is not the case that "the density of $\tilde X$ is given by $\mathcal{L}(x; \theta_{\text{MLE}})$", for example. The density of $\tilde X$ is $N(\mu, \sigma^2/n)$ - but you don't know $\sigma^2/n$, hence the $t$ distribution when you substitute the (random) estimate $s^2$ for $\sigma^2$. $\endgroup$
    – jbowman
    Oct 19, 2021 at 17:23
  • $\begingroup$ Thanks for continuing to help me work through this! Unfortunately, I'm still fairly confused. The likelihood function, after fixing a value of the parameter (i.e., $\theta_{\text{MLE}}$ here) is a density over possible observed data $\tilde X$ (Wikipedia). My choice of notation probably wasn't optimal—$\tilde X$, in the example we're discussing, is meant to denote not a single draw from a univariate normal but a vector of $n$ i.i.d. draws from $\mathcal{N}(\mu_{\text{MLE}}, \sigma_{\text{MLE}})$, ... $\endgroup$
    – jgaeb
    Oct 19, 2021 at 19:21
  • $\begingroup$ ... where $\mu_{\text{MLE}} = \tfrac 1 n \sum_{i = 1}^n X_i$ and $\sigma_{\text{MLE}} = \sqrt{\tfrac 1 n \sum_{i = 1}^n (X_i - \mu_{\text{MLE}})^2}$, where $X = (X_1, \ldots, X_n)$ are the observed data. It's definitely true, as you point out, that the joint distribution of $(\tfrac 1 n \sum_{i = 1}^n \tilde X_i, \sqrt{\tfrac 1 n \sum_{i = 1}^n (\tilde X_i - \mu_{\text{MLE}})^2})$—where $\tilde X = (\tilde X_1, \ldots, \tilde X_n)$ denotes a vector of $n$ i.i.d. $\mathcal{N}(\mu_{\text{\MLE}}, \sigma_{\text{MLE}})$ variables—is two dimensional. ... $\endgroup$
    – jgaeb
    Oct 19, 2021 at 19:21

1 Answer 1

0
$\begingroup$

The issue, I've now realized, is that Steps 1, 2, and 3 are not actually the frequentist definition of a confidence interval. According to Wikipedia, the definition is actually a pair of functions $u$ and $v$ such that $\Pr(u(X) < \theta < v(X)) = 1 - \alpha$. Steps 1, 2, and 3 are just a general way of (approximately) constructing such $u$ and $v$. So, the point is, in this particular case, $$ \Pr(\mu_{\text{MLE}} - t^{n-1}_{\alpha/2} \sigma_{\text{MLE}} < \theta) = \tfrac \alpha 2 $$ while $$ \Pr(\mu_{\text{MLE}} - z_{\alpha/2} \sigma_{\text{MLE}} < \theta) > \tfrac \alpha 2 $$ where $t^{n-1}_{\alpha/2}$ is the $\tfrac \alpha 2$-th quantile for a $t$-distribution with $n-1$ degrees of freedom and $z_{\alpha/2}$ is the $\tfrac \alpha 2$-th quantile for a standard normal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.