# The basic logic of constructing a confidence interval

Consider a model with a parameter of interest, $$\theta$$, and its point estimator, $$\hat\theta$$. For simplicity, assume $$\hat\theta\sim N(\theta,\sigma^2/n)$$ (in numerous instances this could be justified asymptotically). There are two ways of constructing an interval that happens to be the shortest possible $$(1-\alpha)$$ level confidence interval.

1. For any true value $$\theta$$, I want the shortest possible interval $$(\hat\theta_{lower},\hat\theta_{upper})$$ that has $$(1-\alpha)$$ probability of capturing $$\theta$$. I select the highest density region in the distribution of $$\hat\theta$$ given $$\theta$$, $$f(\hat\theta;\theta)$$, so that the cumulative probability for that region is $$(1-\alpha)$$. I define the interval estimator such that for every point estimate $$\hat\theta$$ in the region, the corresponding interval estimate would cover $$\theta$$.
Since the distribution of $$\hat\theta$$ is the same for any true value $$\theta$$ except for a location shift, the mechanism (the rule) for constructing the interval is independent of what the actual $$\theta$$ is. Hence, it will cover any true $$\theta$$ with $$(1-\alpha)$$ probability.

2. Given a point estimate $$\hat\theta$$, I am considering under what true value $$\theta$$ it is likely to have been generated. Knowing the distribution of $$\hat\theta$$ for any given true $$\theta$$, $$f(\hat\theta;\theta)$$, I select those $$\theta$$s that yield the highest density values. I limit the selection to only include values $$\theta$$ that have the cumulative probability $$\geq\alpha$$ for values at least as extreme as $$\theta$$; in other words, the values $$\theta$$ for which the corresponding $$p$$-value associated with $$\hat\theta$$ is at least $$\alpha$$.

The first approach focuses directly on ensuring that whatever the true $$\theta$$, it is included in $$(1-\alpha)$$ share of sampling instances. The second approach looks for the best candidate $$\theta$$s that make the realization $$\hat\theta$$ likely, while discarding $$\theta$$s under which $$\hat\theta$$ is unlikely. The line between the two (likely vs. unlikely) is drawn somewhat arbitrarily from the perspective of the original goal, but it happens to be the right line.

The two rules for constructing an interval give the same answer in this simplified example.
Which (if any of the two) is the correct motivation for, or the correct way of thinking about, the construction of a confidence interval?
(Perhaps removing the distributional assumption for $$\hat\theta$$ above would invalidate one of the approaches, making it clear that it is generally inappropriate and only gives the right answer in this example by coincidence?)

• What (if any) is your motivation here? It seems like a very subtle difference for the vast majority of cases. – Xiaomi Oct 3 '18 at 7:46
• @Xiaomi, thank you for your interest! The result of the two approaches is the same, but the way of arriving at it is apparently very different (at least that is my perception). I wonder which way one should proceed to stay true to the logic (and probably the history) of confidence interval estimation. Perhaps one of the ways would only occasionally give the right answer. I am quite concerned, even though the difference might appear subtle to some. Would you happen to know the answer? – Richard Hardy Oct 3 '18 at 7:51
• After Martijn Weterings's answer I am starting to think that the second approach might be a special case of constructing a credible interval (with a flat prior on $\theta$). – Richard Hardy Oct 3 '18 at 10:34
• Related question about the difference between confidence interval and credible interval with flat prior stats.stackexchange.com/questions/355109/… (they are not the same, and especially noteworthy is that a confidence interval does not change with a change of variables whereas a credible interval, where the prior needs to be changed if you want to keep it 'flat', does not remain the same) – Martijn Weterings Oct 4 '18 at 1:46
• The last sentence of the paragraph explaining the second method "I limit the selection to only include...the values $\theta$ for which the corresponding p-value associated with $\theta$ is at least $\alpha$" is actually very much the same as the first method. – Martijn Weterings Oct 4 '18 at 1:51

## Example with 100 Bernoulli trials

The construction of confidence intervals could be placed in a plot of $$\theta$$ versus $$\hat{\theta}$$ like here:

Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?

In my answer to that question I use the following graph:

Note that this image is a classic and an adaptation from The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial C. J. Clopper and E. S. Pearson Biometrika Vol. 26, No. 4 (Dec., 1934), pp. 404-413

You could define a $$\alpha$$-% confidence region in two ways:

• in vertical direction $$L(\theta) < X < U(\theta)$$ the probability for the data $$X$$, conditional on the parameter being truly $$\theta$$, to fall inside these bounds is $$\alpha$$ .

• in horizontal direction $$L(X) < \theta < U(X)$$ the probability that an experiment will have the true parameter inside the confidence interval is $$\alpha$$%.

## Correspondence between two directions

So the key-point is that there is a correspondence between the intervals $$L(X),U(X)$$ and the intervals $$L(\theta),U(\theta)$$. This is where the two methods come from.

When you want $$L(X)$$ and $$U(X)$$ to be as close as possible ("the shortest possible ($$1−\alpha$$) level confidence interval") then you are trying to make the area of the entire region as small as possible, and this is similar to getting $$L(\theta)$$ and $$U(\theta)$$ as close as possible. (more or less, there is no unique way to get the shortest possible interval, e.g. you can make the interval shorter for one type of observation $$\hat\theta$$ at the cost of another type of observation $$\hat\theta$$)

## Example with $$\boldsymbol{\hat\theta \sim \mathcal{N}(\mu=\theta, \sigma^2=1+\theta^2/3)}$$

To illustrate the difference between the first and second method we adjust the example a bit such that we have a case where the two methods do differ.

Let the $$\sigma$$ not be constant but instead have some relation with $$\mu= \theta$$ $${\hat\theta \sim \mathcal{N}(\mu=\theta, \sigma^2=1+\theta^2/3)}$$

then the probability density function for $$\hat \theta$$, conditional on $$\theta$$ is $$f(\hat\theta, \theta ) = \frac{1}{\sqrt{2 \pi (1+\theta^2/3)}} exp \left[ \frac{-(\theta-\hat\theta)^2}{2(1+\theta^2/3)} \right]$$

Imagine this probability density function $$f(\hat \theta , \theta)$$ plotted as function of $$\theta$$ and $$\hat \theta$$.

Legend: The red line is the upper boundary for the confidence interval and the green line is the lower boundary for the confidence interval. The confidence interval is drawn for $$\pm 1 \sigma$$ (approximately 68.3%). The thick black lines are the pdf (2 times) and likelihood function that cross in the points $$(\theta,\hat\theta)=(-3,-1)$$ and $$(\theta,\hat\theta)=(0,-1)$$.

PDF In the direction from left to right (constant $$\theta$$) we have the pdf for the observation $$\hat \theta$$ given $$\theta$$. You see two of these projected (in the plane $$\theta = 7$$). Note that the $$p$$-values boundaries ($$p<1-\alpha$$ chosen to be the highest density region) are on the same height for a single pdf, but not for not at the same height for different pdf's (by height that means the value of $$f(\hat\theta,\theta)$$)

Likelihood function In the direction from top to bottom (constant $$\hat \theta$$) we have the likelihood function for $$\theta$$ given the observation $$\hat\theta$$. You see one of these projected on the right.

For this particular case, when you select the 68% mass with the highest density for constant $$\theta$$ then you do not get the same as selecting the 68% mass with the highest likelihood for constant $$\hat \theta$$.

For other percentages of the confidence interval you will have one or both of the boundaries at $$\pm \infty$$ and also the interval may consist of two disjoint pieces. So, that is obviously not where the highest density of the likelihood function is (method 2). This is a rather artificial example (although it is simple and nice how it results in these many details) but also for more common cases you get easily that the two methods do not coincide (see the example here where the confidence interval and the credible interval with a flat prior are compared for the rate parameter of a exponential distribution).

## When are the two methods the same?

This horizontal vs vertical is giving the same result, when the boundaries $$U$$ and $$L$$, that bound the intervals in the plot $$\theta$$ vs $$\hat \theta$$ are iso-lines for $$f(\hat \theta ; \theta)$$. If boundaries are everywhere at the same height than in neither of the two directions you can make an improvement.

(contrasting with this: in the example with $$\hat \theta \sim \mathcal{N}(\theta,1+\theta^2/3)$$ the confidence interval boundaries will not be at the same value $$f(\hat \theta, \theta)$$ for different $$\theta$$, because the probability mass becomes more spread out, thus lower density, for larger $$\vert \theta \vert$$. This makes that $$\theta_{low}$$ and $$\theta_{high}$$ will not be at the same value $$f(\hat \theta ; \theta)$$, at least for some $$\hat \theta$$, This contradicts with method 2 that seeks to select the highest densities $$f(\hat \theta ; \theta)$$ for a given $$\hat \theta$$. In the image above I have tried to emphasize this by plotting the two pdf functions that relate to the confidence interval boundaries at the value $$\hat \theta= -1$$; you can see that they have different values of the pdf at these boundaries.)

Actually the second method doesn't seem entirely right (it is more a sort of variant of a likelihood interval or a credible interval than a confidence interval) and when you select $$\alpha$$% density in the horizontal direction (bounding $$\alpha$$ % of the mass of the likelihood function) then you may be dependent on the prior probabilities.

In the example with the normal distribution it is not a problem and the two methods align. For an illustration see also this answer of Christoph Hanck. There the boundaries are iso-lines. When you change the $$\theta$$ the function $$f(\hat\theta,\theta)$$ only makes a shift and does not change 'shape'.

## Fiducial probability

The confidence interval, when the bounds are created in vertical direction, are independent of the prior probabilities. This is not the case with the 2nd method.

This difference between the first and the second method may be a good example of the subtle difference between fiducial probability and confidence intervals.

• Good points. I was suspecting some prior probabilities might be be trying to sneak in in the second approach... – Richard Hardy Oct 3 '18 at 9:41
• I will try and see whether I can come up with some better visual representation. When you would plot the $f(\hat \theta; \theta)$ as a surface then you get some ridge shape, but in the case of the Bernouilli trials that shape is smaller and higher at the edges. In the case of the normal distribution it is more symmetric. – Martijn Weterings Oct 3 '18 at 9:45
• That (a new visual representation) might help a lot! Besides, could you elaborate on Imagine the probability density function $f(\hat \theta ; \theta)$ for $\hat \theta$ conditional on $\theta$ plotted in 2D $\theta$ versus $\hat \theta$? Could you rephrase it somehow? I am having trouble understanding this and then consequently what is meant by horizontal vs. vertical in the remainder; perhaps you could give names like "the $\theta$ direction" for horizontal and "the $\hat\theta$ direction" for vertical (or otherwise, whichever is correct). – Richard Hardy Oct 3 '18 at 10:04
• I would like to add a picture of that. It is similar to the current image. We normally see $f(\hat \theta ; \theta)$ as a function with $\theta$ fixed but we could change it into a function $f(\hat \theta, \theta)$ with $\theta$ not fixed. Then when we make a confidence interval we create boundaries $L(\theta)$ and $U(\theta)$ that bound, in vertical ($\hat \theta$) direction, $\alpha$% of the mass. Since we do this for every $\theta$ we will in the 2D image have boundaries that contain $\alpha$% of the mass. We could imagine doing the same in the other direction (but it will be different). – Martijn Weterings Oct 3 '18 at 10:08
• @whuber the image is pretty old, I used the image from here jstor.org/stable/2331986 The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial C. J. Clopper and E. S. Pearson Biometrika Vol. 26, No. 4 (Dec., 1934), pp. 404-413 (and I agree that the idea of choosing an area such that you get 95% of the mass is not correct, only one of those regions will result in confidence intervals, the problem might be in the statement of the question "I want the shortest possible interval" which is ambiguous. There is no single unique way to get this.) – Martijn Weterings Oct 3 '18 at 20:53