The basic logic of constructing a confidence interval

Question

Consider a model with a parameter of interest, $\theta$, and its point estimator, $\hat\theta$. For simplicity, assume $\hat\theta\sim N(\theta,\sigma^2/n)$ (in numerous instances this could be justified asymptotically). There are two ways of constructing an interval that happens to be the shortest possible $(1-\alpha)$ level confidence interval.

1. For any true value $\theta$, I want the shortest possible interval $(\hat\theta_{lower},\hat\theta_{upper})$ that has $(1-\alpha)$ probability of capturing $\theta$. I select the highest density region in the distribution of $\hat\theta$ given $\theta$, $f(\hat\theta;\theta)$, so that the cumulative probability for that region is $(1-\alpha)$. I define the interval estimator such that for every point estimate $\hat\theta$ in the region, the corresponding interval estimate would cover $\theta$.
   Since the distribution of $\hat\theta$ is the same for any true value $\theta$ except for a location shift, the mechanism (the rule) for constructing the interval is independent of what the actual $\theta$ is. Hence, it will cover any true $\theta$ with $(1-\alpha)$ probability.

2. Given a point estimate $\hat\theta$, I am considering under what true value $\theta$ it is likely to have been generated. Knowing the distribution of $\hat\theta$ for any given true $\theta$, $f(\hat\theta;\theta)$, I select those $\theta$s that yield the highest density values. I limit the selection to only include values $\theta$ that have the cumulative probability $\geq\alpha$ for values at least as extreme as $\theta$; in other words, the values $\theta$ for which the corresponding $p$-value associated with $\hat\theta$ is at least $\alpha$.

The first approach focuses directly on ensuring that whatever the true $\theta$, it is included in $(1-\alpha)$ share of sampling instances. The second approach looks for the best candidate $\theta$s that make the realization $\hat\theta$ likely, while discarding $\theta$s under which $\hat\theta$ is unlikely. The line between the two (likely vs. unlikely) is drawn somewhat arbitrarily from the perspective of the original goal, but it happens to be the right line.

The two rules for constructing an interval give the same answer in this simplified example.
Which (if any of the two) is the correct motivation for, or the correct way of thinking about, the construction of a confidence interval?
(Perhaps removing the distributional assumption for $\hat\theta$ above would invalidate one of the approaches, making it clear that it is generally inappropriate and only gives the right answer in this example by coincidence?)

Answer 1

Example with 100 Bernoulli trials

The construction of confidence intervals could be placed in a plot of $\theta$ versus $\hat{\theta}$ like here:

Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?

In my answer to that question I use the following graph:

Note that this image is a classic and an adaptation from The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial C. J. Clopper and E. S. Pearson Biometrika Vol. 26, No. 4 (Dec., 1934), pp. 404-413

You could define a $\alpha$-% confidence region in two ways:

• in vertical direction $L(\theta) < X < U(\theta)$ the probability for the data $X$, conditional on the parameter being truly $\theta$, to fall inside these bounds is $\alpha$ .

• in horizontal direction $L(X) < \theta < U(X)$ the probability that an experiment will have the true parameter inside the confidence interval is $\alpha$%.

---

Correspondence between two directions

So the key-point is that there is a correspondence between the intervals $L(X),U(X)$ and the intervals $L(\theta),U(\theta)$. This is where the two methods come from.

When you want $L(X)$ and $U(X)$ to be as close as possible ("the shortest possible ($1−\alpha$) level confidence interval") then you are trying to make the area of the entire region as small as possible, and this is similar to getting $L(\theta)$ and $U(\theta)$ as close as possible. (more or less, there is no unique way to get the shortest possible interval, e.g. you can make the interval shorter for one type of observation $\hat\theta$ at the cost of another type of observation $\hat\theta$)

---

Example with $\boldsymbol{\hat\theta \sim \mathcal{N}(\mu=\theta, \sigma^2=1+\theta^2/3)}$

To illustrate the difference between the first and second method we adjust the example a bit such that we have a case where the two methods do differ.

Let the $\sigma$ not be constant but instead have some relation with $\mu= \theta$ $${\hat\theta \sim \mathcal{N}(\mu=\theta, \sigma^2=1+\theta^2/3)}$$

then the probability density function for $\hat \theta$, conditional on $\theta$ is $$f(\hat\theta, \theta ) = \frac{1}{\sqrt{2 \pi (1+\theta^2/3)}} exp \left[ \frac{-(\theta-\hat\theta)^2}{2(1+\theta^2/3)} \right] $$

Imagine this probability density function $f(\hat \theta , \theta)$ plotted as function of $\theta$ and $\hat \theta$.

Legend: The red line is the upper boundary for the confidence interval and the green line is the lower boundary for the confidence interval. The confidence interval is drawn for $\pm 1 \sigma$ (approximately 68.3%). The thick black lines are the pdf (2 times) and likelihood function that cross in the points $(\theta,\hat\theta)=(-3,-1)$ and $(\theta,\hat\theta)=(0,-1)$.

PDF In the direction from left to right (constant $\theta$) we have the pdf for the observation $\hat \theta$ given $\theta$. You see two of these projected (in the plane $\theta = 7$). Note that the $p$-values boundaries ($p<1-\alpha$ chosen to be the highest density region) are on the same height for a single pdf, but not for not at the same height for different pdf's (by height that means the value of $f(\hat\theta,\theta)$)

Likelihood function In the direction from top to bottom (constant $\hat \theta$) we have the likelihood function for $\theta$ given the observation $\hat\theta$. You see one of these projected on the right.

For this particular case, when you select the 68% mass with the highest density for constant $\theta$ then you do not get the same as selecting the 68% mass with the highest likelihood for constant $\hat \theta$.

For other percentages of the confidence interval you will have one or both of the boundaries at $\pm \infty$ and also the interval may consist of two disjoint pieces. So, that is obviously not where the highest density of the likelihood function is (method 2). This is a rather artificial example (although it is simple and nice how it results in these many details) but also for more common cases you get easily that the two methods do not coincide (see the example here where the confidence interval and the credible interval with a flat prior are compared for the rate parameter of a exponential distribution).

When are the two methods the same?

This horizontal vs vertical is giving the same result, when the boundaries $U$ and $L$, that bound the intervals in the plot $\theta$ vs $\hat \theta$ are iso-lines for $f(\hat \theta ; \theta)$. If boundaries are everywhere at the same height than in neither of the two directions you can make an improvement.

(contrasting with this: in the example with $\hat \theta \sim \mathcal{N}(\theta,1+\theta^2/3)$ the confidence interval boundaries will not be at the same value $f(\hat \theta, \theta)$ for different $\theta$, because the probability mass becomes more spread out, thus lower density, for larger $\vert \theta \vert$. This makes that $\theta_{low}$ and $\theta_{high}$ will not be at the same value $f(\hat \theta ; \theta)$, at least for some $\hat \theta$, This contradicts with method 2 that seeks to select the highest densities $f(\hat \theta ; \theta)$ for a given $\hat \theta$. In the image above I have tried to emphasize this by plotting the two pdf functions that relate to the confidence interval boundaries at the value $\hat \theta= -1$; you can see that they have different values of the pdf at these boundaries.)

Actually the second method doesn't seem entirely right (it is more a sort of variant of a likelihood interval or a credible interval than a confidence interval) and when you select $\alpha$% density in the horizontal direction (bounding $\alpha$ % of the mass of the likelihood function) then you may be dependent on the prior probabilities.

In the example with the normal distribution it is not a problem and the two methods align. For an illustration see also this answer of Christoph Hanck. There the boundaries are iso-lines. When you change the $\theta$ the function $f(\hat\theta,\theta)$ only makes a shift and does not change 'shape'.

Fiducial probability

The confidence interval, when the bounds are created in vertical direction, are independent of the prior probabilities. This is not the case with the 2nd method.

This difference between the first and the second method may be a good example of the subtle difference between fiducial/confidence distribution $\frac{d}{d\theta}F(\hat\theta,\theta)$ and the likelihood function $\frac{d}{d\hat\theta}F(\hat\theta,\theta)$ (where $F$ is the cumulative distribution function).