Interpretation of distribution that appears when calculating CI for population mean

Question

Let $X \sim \mathcal{N}(\mu, \sigma)$ be the model for a normally distributed population, described by the probability density function $f_{X}(x; \mu, \sigma)$. We can denote $\mathbf{X} = (X_1, X_2, \ldots, X_n)$ a random sample of size $n$, and $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ is a particular sample.

We know the sample mean $\overline{\mathbf{x}} = \mathbf{Mean}(\mathbf{x})$ is an estimate of the population mean $\mu$. The sampling distribution of the sample mean is a random variable $\overline{\mathbf{X}} = \mathbf{Mean}(\mathbf{X})$, which describes the distribution of possible means we could observe from random samples of size $n$ from the population. I will denote the probability density function of the random variable $\overline{\mathbf{X}}$ as $f_{\overline{\mathbf{X}}|\mu,\sigma}$, in order to show explicitly the dependence on the population parameters $\mu$ and $\sigma$.

CASE 0

Before we get to the confidence interval question about an unknown population mean $\mu$, I want to give an example of a calculation involving the sampling distribution $f_{\overline{\mathbf{X}}|\mu,\sigma}$. When $\mu$ and $\sigma$ are known, the CLT tells us that $\overline{\mathbf{X}} \sim \mathcal{N}(\mu, \frac{\sigma}{\sqrt{n}})$, in other words $f_{\overline{\mathbf{X}}|\mu,\sigma}$ is the pdf of a normal distribution with mean $\mu$ and standard deviation $\mathbf{se} = \frac{\sigma}{\sqrt{n}}$. Using the CLT, we can calculate a confidence interval for the sample means we would expect to observe for samples of size $n$ by choosing appropriate values $a$ and $b$ such that $\Pr(\{a \leq \overline{\mathbf{X}} \leq b\} = \int_{\overline{\mathbf{x}}=a}^{\overline{\mathbf{x}}=b} f_{\overline{\mathbf{X}}|\mu,\sigma}(\overline{\mathbf{x}}|\mu,\sigma)\,d\overline{\mathbf{x}}$ equals to the desired coverage probability.

CASE 1

Now let's tackle the slightly more interesting case we know$\mu$ but $\sigma$ is unknown. We don't know $\sigma^2$ but we can use the sample variance estimate $s^2_{\mathbf{x}} = \mathbf{Var}(\mathbf{x})$ for it. Using the sample standard deviation $s_{\mathbf{x}}$ as an estimate for $\sigma$, we can also obtain an estimated standard error $\widehat{\mathbf{se}} = \frac{s_{\mathbf{x}}}{\sqrt{n}}$. Since we know that sample variances tend to underestimate the population variance, we can't just plug in these estimates into the normal model suggested by the CLT, but instead use Gosset's model based on Student's $t$-distribution. Using a combination of the plug-in principle and Gosset's ``heavy-tail'' correction to compensate for the variability underestimation that we can expect to occur, we obtain the following model for the sampling distribution of the sample mean we can expect to observe: $$ \overline{\mathbf{X}} \sim \mathcal{T}(\texttt{df}=n-1, \texttt{loc}=\mu, \texttt{scale}=\widehat{\mathbf{se}}) = \widehat{\mathbf{se}}\cdot \mathcal{T}(\nu) + \mu, $$ where $\mathcal{T}(\nu)$ is Student's $t$-distribution with $\nu = \texttt{df} = n-1$ degrees of freedom. This result is usually presented in terms of the location-scale pivot: $$ T = \frac{\overline{\mathbf{X}} - \mu}{ \widehat{\mathbf{se}} } \sim \mathcal{T}(n-1), $$ where $\widehat{\mathbf{se}} = \frac{s_{\mathbf{x}}}{\sqrt{n}}$ is the standard error computed estimated from the random sample $\mathbf{x}$.

CASE 2

We now come to the most realistic scenario when both $\mu$ and $\sigma$ are unknown. If plug in the estimate $\overline{\mathbf{x}}$ instead of $\mu$, and $s^2_{\mathbf{x}}$ instead of $\sigma^2$ into Student's $t$-distribution we end up with the following model: $$ \widetilde{\mathbf{X}} \sim \mathcal{T}(\texttt{df}=n-1, \texttt{loc}=\overline{\mathbf{x}}, \texttt{scale}=\widehat{\mathbf{se}}) = \widehat{\mathbf{se}}\cdot \mathcal{T}(\nu) + \overline{\mathbf{x}}, $$ Is there some interpretation we can give to $\widetilde{\mathbf{X}}$ (shown in blue in the figure) ? It's clearly showing something useful---estimating the variability of the sample via $\widehat{\mathbf{se}}$, and centred at the observed mean $\overline{\mathbf{x}}$ since this is our best guess for $\mu$, but I have not seen any stats book that discuss this. The reason for my question is because $\widetilde{\mathbf{X}}$ is also the bootstrap distribution of the mean we from the sample $\mathbf{x}$, as shown with the blue histogram. Is the blue curve showing something like a "frequentist posterior" of the population mean $\mu$? Or the likelihood of $\mu$ given the data?

The interpretation of the blue curve as a likelihood of $\mu$ given data $\mathbf{x}$ seems plausible, Recall the pivotal quantity from CASE 1 $$ T = \frac{\overline{\mathbf{X}} - \mu}{ \widehat{\mathbf{se}} } \sim \mathcal{T}(n-1). $$ If we want to construct a confidence interval for the unknown population mean, we star with $t_\ell$ and $t_u$ which are the 5th and 95th percentiles of the T distribution, which means $\textrm{Pr}( \{ t_\ell \leq T \leq t_u \}) = 0.9$, and after some manipulations end up with $$ Pr( \{ \bar{\mathbf{X}}+t_\ell \cdot \widehat{\mathbf{SE}} \leq \mu \leq \bar{\mathbf{X}}+t_u \cdot \widehat{\mathbf{SE}} \} )= 0.9. $$ which is statement that works for random samples $\mathbf{X}$, which we can then apply for a particular sample by saying $[\bar{\mathbf{x}}+t_\ell \cdot \hat{\mathbf{se}}, \bar{\mathbf{x}}+t_u \cdot \hat{\mathbf{se}} ]$ is a 90% confidence interval for the population mean.

Clearly the confidence interval seems to have been constructed from the model $\widetilde{\mathbf{X}}$ we defined above, but the "inverting the pivotal quantity" procedure didn't provide us with any interpretation for it, since we just followed the procedure mechanically.

So to summarize:

• Q1: what is the correct way to interpret the model $\widetilde{\mathbf{X}}$ (blue in figure)?
• Q2: is the distribution of $\widetilde{\mathbf{X}}$ the same as the one used for inverting-the-pivot procedure for confidence intervals?

Here is the notebook I used to generate the figure: online or mybinder.

Answer 1

I have recently found an excellent paper titled "Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review" by Xie and Singh: https://statweb.rutgers.edu/mxie/RCPapers/insr.12000.pdf which explains clearly the notion of a confidence distribution, which is the term that describes the probability distribution in CASE 2.

The key insight of the paper is that the confidence distribution is viewed as an estimator for the parameter of interest, instead of an inherent distribution of the parameter.

The authors make it clear there is no need to describe $\mu$ as a random variable (Bayesian paradigm), and instead we can talk about $\widetilde{\mathbf{X}}$ entirely within the frequentist paradigm. Just like a point estimate, or a confidence interval, the confidence distribution is a frequentist estimator for the unknown population mean.

Answer 2

Okay I think $\widetilde{\mathbf{X}}$ can be interpreted as the likelihood function $L(\mu)$ after a partial-application of the plug-in principle to match the first moment of the distribution.

Here is the argument. We're starting from the probability model in CASE 1: $$ \overline{\mathbf{X}} = p(\overline{\mathbf{x}} |\mu,\sigma) = Cg\big( (\overline{\mathbf{x}}-\mu)^2 \big)^e + \mu = \widehat{\mathbf{se}}\cdot \mathcal{T}(\nu) + \mu $$ where I have hidden the details of Student's t-distribution behind the constant $C$, funciton $g$, and exponent $e$.

We know $\mathbb{E}(\overline{\mathbf{X}}) = \mu$, and we have observed $\mathbf{Mean}(\mathbf{x}) = \overline{\mathbf{x}}$, which is an estimate of the population mean $\mu$.

We want to update our model to match the observation $\overline{\mathbf{x}}$. Specifically we want to choose a model whose first moment matches the observed data. Currently $\mathbb{E}(\overline{\mathbf{X}}) = \mu$ but we want a new model such that $\mathbb{E}(\widetilde{\mathbf{X}}) = \overline{\mathbf{x}}$, which we can easily do if we define $$ \widetilde{\mathbf{X}} = \overline{\mathbf{X}} \;\;\; + \overline{\mathbf{x}} -\mu, $$ where we adding the term $\overline{\mathbf{x}}-\mu$ to match the first momen of the model to observed data. This horizontal shift doesn't affect the other part of the formula since it has zero mean. We thus obtain the model: $$ \widetilde{\mathbf{X}} = p(\widetilde{\mathbf{x}} |\mu,\sigma) = Cg\big( (\overline{\mathbf{x}}-\mu)^2 \big)^e + \mu \;\; + \overline{\mathbf{x}} - \mu = \widehat{\mathbf{se}}\cdot \mathcal{T}(\nu) + \overline{\mathbf{x}} $$

Let us now consider the function that defines the probability density of the random variable $\widetilde{\mathbf{X}}$ as a likelihood, meaning $\overline{\mathbf{x}}$ is a fixed quantity, and $\mu$ is the variable: $$ L(\mu) = p(\widetilde{\mathbf{x}} |\mu,\sigma) = \widehat{\mathbf{se}}\cdot \mathcal{T}(\nu) + \overline{\mathbf{x}}. $$

The fact that $L(\mu)$ happens to be well normalized is an accidental feature due to the symmetry of $g$, but since we're in frequentist context we are not interpreting $L(\mu)$ as a pdf.

We can therefore understand the construction of the 90% confidence interval for the population mean, $[\bar{\mathbf{x}}+t_\ell \cdot \widehat{\mathbf{se}}, \bar{\mathbf{x}}+t_u \cdot \widehat{\mathbf{se}} ]$ as a procedure for selecting a range of $\mu$s that contains 0.9 proportion of the area under the likelihood function $L(\mu)$.