Confidence Intervals: how to formally deal with $P(L(\textbf{X}) \leq \theta, U(\textbf{X})\geq\theta) = 1-\alpha$

Question

I was thinking about the formal definition of confidence intervals.

Given a Random Sample $\textbf{X} = X_1,X_2,\dots,X_n$ a level $\alpha$ confidence interval for the population parameter $\theta$ is defined as a pair of estimators (function of $\textbf{X}$), namely $L(\textbf{X})$ and $U(\textbf{X})$ with $L(\textbf{X}) \leq U(\textbf{X})$, with the property: $$P(L(\textbf{X}) \leq \theta \leq U(\textbf{X})) = 1-\alpha$$

My interpretation of this this last equality is the joint probability of $L(\textbf{X})$ and $U(\textbf{X})$, i.e.

$$P(L(\textbf{X}) \leq \theta, U(\textbf{X})\geq\theta) = 1-\alpha$$

My question is how to work with this expression to find $L(\textbf{X})$ and $U(\textbf{X})$?

If I call $f_{L,U}(l,u)$ the joint pdf of $L(\textbf{X})$ and $U(\textbf{X})$ it should be something like:

$$\int_{- \infty}^{\theta}\int_{\theta}^{+\infty}f_{L,U}(l,u)dl \ du = 1 - \alpha$$

and then I am stucked. I don't know how to go on.

My first question is: since both $L(\textbf{X})$ and $U(\textbf{X})$ are function of the same $\textbf{X}$ does something like the joint density even make sense?

I don't know if my calculation is right but I found that something similar to a joint CDF for $L(\textbf{X})$ and $U(\textbf{X})$ could be (if $X \in \rm {I\!R} $)

$$F_{L,U}(l,u)= F_{\textbf{X}}\bigg(min\big(L^{-1}(l),U^{-1}(u)\big)\bigg)$$ if that makes any sense at all. What is the correct way to think about this? $$-----------------------------------$$ I know that there are methods like pivotal quantities but the difference with respect to my case is that when we have Pivot the probabilistic statement is expressed it terms of only one random variable so I don't have a joint density.

Suppose that I call $Q(\textbf{X},\theta)$ my pivot, I can find say $l$ and $u$ such that: $$P\big(l\leq Q(\textbf{X},\theta) \leq u \big) = 1 - \alpha$$ and then I can invert this relationship wrt $\theta$. I suppose this equality could be rewritten as:

$$F_{Q}(u) - F_{Q}(l) = 1 - \alpha$$

This last equation has infinite solutions since I have 2 unknowns $l$ and $u$. My understanding is that, in order to solve it, one has to choose value for one between $l$ and $u$. For instance I could choose $l$ such that $F_{Q}(l) = 2\%$ and solve fo $u$ so that $F_{Q}(u) = 1 - \alpha + 2\%$

And here is my second question: so there the $level- \alpha$ confidence intervals are infinite? And the difference between them is their length? Is my reasoning correct?

Any help would be much appreciated! Thank you.

Answer 1

Possibly a Clopper-Pearson interval may help to obtain an intuition about these confidence intervals. (the below is a variation of an answer to How to estimate a probability of an event to occur based on its count? more specifically it is a variation of a graph from Clopper-Pearson )

The main trick here is that we can switch L and U from being functions of X to being functions of $\mathbb{\theta}$

Imagine the case of 100 Bernoulli trials where the probability of success is $\theta$ and we observe the total number of successes $X$.

When we observe an $X$ as if it came from the unknown population of Bernoulli trials with true (unknown) probability $\theta$ then we will choose $U(X)$ and $L(X)$ such that no matter what the the real $\theta$ the probability to make a mistake is $\alpha$ in estimating $U(X)$ and L(X).

• This occurs when we select for a given $X$ a confidence interval for $\theta$ (based on $L(x)$ and $U(x)$) such that $X$ occurs in a fraction $1-\alpha$ of all the possible cases of $\theta$ in the confidence interval (based on $L(\theta)$ and $U(\theta)$). There is some degree of freedom in shifting more or less weight between $U$ and $L$ and there are many different ways to do this.

• If we do this consistently every-time that we perform an experiment, then in a fraction $1-\alpha$ of the cases we will observe an $X$ that let's us include the true $\theta$ inside the interval and in a fraction $\alpha$ of the cases we will not include $\theta$ inside the interval.

  (this is depicted in the image by the colored lines for the case $\theta=0.2$, the gray lines are cases when we select the right interval, the red when the interval is too high and the green when the interval is too low.)

$$-----------------------------------$$

More formally:

if we choose a confidence interval such that

$$I_{\alpha}(X) = \lbrace \theta: F_X(\alpha/2,\theta) \leq X \leq F_X(1-\alpha/2,\theta) \rbrace$$

then we have a $1-\alpha$ confidence interval.

The above means that we choose, for a given observation $x$, those $\theta$ into the interval for which the observation $x$ would occur within a $1-\alpha$ interval $P( L<x<U \vert \theta) = 1-\alpha$, where the $L$ and $U$ are now functions of $\theta$.

Then given any real $\theta$ we will observe:

• a fraction $1 - \alpha$ of the time an $X$ such that $$F_X(\alpha/2,\theta) \leq X \leq F_X(1-\alpha/2,\theta)$$
• and a fraction $\alpha$ of the time an $X$ such that $$X < F_X(\alpha/2,\theta) \text{ or } X > F_X(1-\alpha/2,\theta)$$

If you like you could write out the $f_{L,U|\theta}$ which relates to $f_{X|\theta}$ and L(X) and U(X) (both L(X) and U(X), functions of X, are indeed correlated).

But the figure above already shows enough, e.g. if $\theta=0.20$ then in 0+1+3+12+45+140 cases the $\theta<L(X)$ and in 196+48+8+1+0 cases the $\theta>U(X)$ while in 358+755+1297+1795+1974+1697+1119+551 cases $L(X) \leq \theta \leq U(X)$

$$-----------------------------------$$

If the shape of $x_U=U(\theta)$ and $x_L=L(\theta)$ is convex (like in the figure above), then we can use the inverses of those functions.

Answer 2

The following explanation is taken from Puza and O'Neill (2008), which looks at optimal confidence intervals via tail functions. Suppose you have a pivotal quantity $Q(X, \theta)$ for an unknown parameter $\theta \in \Theta$, and let $F_Q$ denote the distribution function for this quantity (which by definition does not depend on $\theta$). Assuming that the pivotal quantity is continuous we have $F_Q(Q(X,\theta)) \sim \text{U}(0,1)$. Now, let $\tau : \Theta \rightarrow [0,1]$ be some function that maps the parameter to a number in the unit interval. Then for all $0< \alpha <1$ and all $\theta \in \Theta$ you have:

$$1-\alpha = \mathbb{P} \Big( \alpha \tau(\theta) \leqslant F_Q(Q(X, \theta)) \leqslant 1-\alpha + \alpha \tau(\theta) \Big).$$

A confidence interval can be obtained from this probability interval by the following inversion:

$$\begin{array}{ll} U(x) = \theta & \text{solves} & Q(x, \theta) = F_Q^{-1} (\alpha \tau(\theta)), \\[6pt] L(x) = \theta & \text{solves} & Q(x, \theta) = F_Q^{-1} (1-\alpha + \alpha \tau(\theta)). \end{array}$$

This inversion can be solved analytically in some cases, and numerically in others. The tail function $\tau$ controls the tails of the probability interval that is used to derive the confidence interval. There are an infinite number of confidence intervals corresponding to the possible choices of tail functions controlling these tails. Manipulating this function lets you adjust the interval you want to "optimise" your analysis (e.g., minimise interval length or meet some other optimisation criterion).

---

Example (normal value with unit variance): Suppose you have $X \sim \text{N}(\theta, 1)$ and you want a confidence interval for $\theta$. Taking the pivotal quantity $Q(X, \theta) = X - \theta \sim \text{N}(0,1)$ gives the standard normal distribution function $F_Q = \Phi$. Hence, the general interval is formed by taking:

$$\begin{array}{ll} x - U(x) = \Phi^{-1} (\alpha \tau(U(x))), \\[6pt] x - L(x) = \Phi^{-1} (1-\alpha + \alpha \tau(L(x))), \end{array}$$

which gives:

$$\begin{array}{ll} U(x) = x - \Phi^{-1} (\alpha \tau(U(x))), \\[6pt] L(x) = x + \Phi^{-1} (1-\alpha + \alpha \tau(L(x))). \end{array}$$

Taking $\tau(\theta) = \tfrac{1}{2}$ gives the symmetric bounds:

$$\begin{array}{ll} U(x) = x - \Phi^{-1} (\alpha / 2 ) = x + z_{\alpha/2}, \\[6pt] L(x) = x - \Phi^{-1} (1- \alpha / 2 ) = x - z_{\alpha/2}, \end{array}$$ where $z_{\alpha/2}$ is the tail probability for the standard normal distribution. This gives the standard symmetric z-based confidence interval for the mean parameter of a normal distribution.

---

We are now in a position to answer your specific questions: (1) Since the lower and upper bound functions are both functions of the same underlying value $X$, their joint density will be a ridge along the pairs of corresponding values that occur for values of the random variable; this is not very useful; (2) there are an infinite number of confidence intervals that can be formed by this method, corresponding to trade-offs in the tail probabilities on the two sides of the probability interval.