Some theory on optimal confidence intervals
Confidence intervals are formed from pivotal quantities, which are functions of the data and parameter of interest that have a distribution that does not depend on the parameters of the problem. Confidence "intervals" are a special case of the broader class of confidence sets, which need not be connected intervals. However, for the purposes of simplicity, we will restrict the present answer to cases where the confidence set is a single interval (i.e., a confidence interval).
Suppose we want to form a confidence interval for the unknown parameter $\phi$ at confidence level $1-\alpha$ using the data $\mathbf{x}$. Consider a continuous pivotal quantity $H(\mathbf{x}, \phi)$ with a distribution that has quantile function $Q_H$. (Note that this function does not depend on the parameter $\phi$ or the data since it is a pivotal quantity.) Using the pivotal quantity, we can choose any value $0 \leqslant \theta \leqslant \alpha$ and form a probability interval from the quantile function. We then "invert" the inequality expression to turn this into an interval statement for the parameter of interest:
$$\begin{align}
1-\alpha
&= \mathbb{P}(Q_H(\theta) \leqslant H(\mathbf{X}, \phi) \leqslant Q_H(1-\alpha+\theta)) \\[6pt]
&= \mathbb{P}(L_\mathbf{X}(\alpha, \theta) \leqslant \phi \leqslant U_\mathbf{X}(\alpha, \theta)). \\[6pt]
\end{align}$$
Substituting the observed data $\mathbf{x}$ then gives the general form for the confidence interval:
$$\text{CI}_\phi(1-\alpha) \equiv \Big[ L_\mathbf{x}(\alpha, \theta), U_\mathbf{x}(\alpha, \theta) \Big].$$
The functions $L_\mathbf{x}$ and $U_\mathbf{x}$ are lower and upper bound functions for the interval, and they depend on the confidence level for the interval and our choice of $\theta$. This latter parameter represents the left tail area used in the initial probability interval for the pivotal quantity, and it can be varied over the above range. If we want to form the optimal (shortest) confidence interval at the confidence level $1-\alpha$, we need to solve the following optimisation problem:
$$\underset{0 \leqslant \theta \leqslant \alpha}{\text{Minimise}} \ \text{Length}(\theta)
\quad \quad \quad \quad \quad
\text{Length}(\theta) \equiv U_\mathbf{x}(\alpha, \theta) - L_\mathbf{x}(\alpha, \theta)$$
Generally speaking, the minimising value $\hat{\theta}$ will depend on the data $\mathbf{x}$ and the value $\alpha$ determining the confidence level. The length of the resulting optimal (shortest) confidence interval will likewise depend on the data and the confidence level. We will see below that in some cases the optimising point does not depend on the data values at all, but even in this case the resulting length of the optimised interval depends on the data and confidence level (just as you would expect).
In problems involving a continuous pivotal quantity, this optimisation can usually be solved using standard calculus method. (And thankfully, for some intervals the work has already been done for you in some functions in the stat.extend package.) Below we give some examples looking at confidence intervals for the population mean and standard deviation for normal data. Assuming that the optimisation part leads to a minimising value for all data values, this will give you a confidence interval that is the shortest interval formed from inversion of the initial pivotal quantity. We will also show how to compute these intervals directly from existing R functions. It is important to note that there will be other confidence intervals formed with other methods that may be shorter for particular samples.$^\dagger$
Example 1 (CI of population mean for normal data): Suppose we observe data $X_1,...,X_n \sim \text{IID N}(\mu, \sigma^2)$ known to come from a normal distribution with unknown parameters. In order to form a CI for the mean parameter $\mu$ we can use the well-known pivotal quantity:
$$\sqrt{n} \cdot \frac{\bar{X}_n - \mu}{S_n} \sim \text{St}(n-1).$$
Suppose we let $t_{n-1, \alpha}$ denote the critical point of the T-distribution with $n-1$ degrees-of-freedom and with upper tail $\alpha$. Using the above pivotal quantity, and choosing any value $0 \leqslant \theta \leqslant \alpha$, we have:
$$\begin{align}
1-\alpha
&= \mathbb{P} \Bigg( -t_{n-1, \theta} \leqslant \sqrt{n} \cdot \frac{\bar{X}_n - \mu}{S_n} \leqslant t_{n-1, \alpha-\theta} \Bigg) \\[6pt]
&= \mathbb{P} \Bigg( \bar{X}_n - \frac{t_{n-1, \alpha-\theta}}{\sqrt{n}} \cdot S_n \leqslant \mu \leqslant \bar{X}_n + \frac{t_{n-1, \theta}}{\sqrt{n}} \cdot S_n \Bigg), \\[6pt]
\end{align}$$
giving the confidence interval:
$$\text{CI}_\mu(1-\alpha) = \Bigg[ \bar{x}_n - \frac{t_{n-1, \alpha-\theta}}{\sqrt{n}} \cdot s_n , \ \bar{x}_n + \frac{t_{n-1, \theta}}{\sqrt{n}} \cdot s_n \Bigg],$$
with length function:
$$\text{Length}(\theta) = ( t_{n-1, \alpha-\theta} + t_{n-1, \theta}) \cdot \frac{s_n}{\sqrt{n}}.$$
In order to minimise this function, we can observe that the critical point function is a convex function of its tail area, which means that the length function is maximised at the point where the upper tail areas in the two parts are the same. (I leave it to the reader to perform the relevant calculus steps to demonstrate this.) This gives the solution:
$$\alpha - \hat{\theta} = \hat{\theta}
\quad \quad \implies \quad \quad
\hat{\theta} = \frac{\alpha}{2}.$$
Thus, we can confirm that the optimal (shortest) confidence interval in this case is the symmetric confidence interval:
$$\text{CI}_\mu(1-\alpha) = \Bigg[ \bar{x}_n \pm \frac{t_{n-1, \alpha/2}}{\sqrt{n}} \cdot s_n \Bigg].$$
In this particular case, we see that the standard symmetric interval (with each tail area the same) is the optimal confidence interval. Varying the relative tail areas away from equal areas increases the length of the interval and so it is not advisable. This standard confidence interval can be programmed using the CONF.mean function in the stat.extend package.
#Generate some data
set.seed(1)
n <- 60
MEAN <- 12
SDEV <- 3
DATA <- rnorm(n, mean = MEAN, sd = SDEV)
#Compute 95% confidence interval for the mean
library(stat.extend)
CONF.mean(alpha = 0.05, x = DATA)
Confidence Interval (CI)
95.00% CI for mean parameter for infinite population
Interval uses 60 data points from data DATA with sample variance = 6.5818
and assumed kurtosis = 3.0000
[10.6225837668173, 14.0231144933285]
Example 2 (CI of population standard deviation for normal data): Continuing the above problem, suppose we now want to form a CI for the standard deviation parameter $\sigma$. To do this we can use the well-known pivotal quantity:
$$\sqrt{n-1} \cdot \frac{S_n}{\sigma} \sim \text{Chi}(n-1).$$
Suppose we let $\chi_{n-1, \alpha}$ denote the critical point of the chi distribution with $n-1$ degrees-of-freedom and with upper tail $\alpha$. Using the above pivotal quantity, and choosing any value $0 \leqslant \theta \leqslant \alpha$, we have:
$$\begin{align}
1-\alpha
&= \mathbb{P} \Bigg( \chi_{n-1, \theta} \leqslant \sqrt{n-1} \cdot \frac{S_n}{\sigma} \leqslant \chi_{n-1, 1-\alpha+\theta} \Bigg) \\[6pt]
&= \mathbb{P} \Bigg( \frac{\sqrt{n-1} \cdot S_n}{\chi_{n-1, 1-\alpha+\theta}} \leqslant \sigma \leqslant \frac{\sqrt{n-1} \cdot S_n}{\chi_{n-1, \theta}} \Bigg), \\[6pt]
\end{align}$$
giving the confidence interval:
$$\text{CI}_{\sigma}(1-\alpha) = \Bigg[ \frac{\sqrt{n-1} \cdot s_n}{\chi_{n-1, 1-\alpha+\theta}}, \ \frac{\sqrt{n-1} \cdot s_n}{\chi_{n-1, \theta}} \Bigg],$$
with length function:
$$\text{Length}(\theta) = \Bigg( \frac{1}{\chi_{n-1, \theta}} - \frac{1}{\chi_{n-1, 1-\alpha+\theta}} \Bigg) \cdot \sqrt{n-1} \cdot s_n.$$
This function can be minimised numerically to yield the minimising value $\hat{\theta}$, which gives the optimal (shortest) confidence interval for the population standard deviation. Unlike in the case of a confidence interval for the population mean, the optimal interval in this case does not have equal tail areas for the upper and lower tail. This problem is examined in Tate and Klett (1959), where the authors look at the corresponding interval for the population variance.
This confidence interval can be programmed using the CONF.var function in the stat.extend package.
#Compute 95% confidence interval for the variance
CONF.var(alpha = 0.05, x = DATA, kurt = 3)
Confidence Interval (CI)
95.00% CI for variance parameter for infinite population
Interval uses 60 data points from data DATA with sample variance = 6.5818
and assumed kurtosis = 3.0000
Computed using nlm optimisation with 8 iterations (code = 3)
[4.50233916286611, 9.41710949707062]
$^\dagger$ To see this, suppose you have a parameter $\theta \in \Theta$ and consider the class of confidence intervals constructed as follows. Choose some event $Y \in \mathscr{Y}$ using an exogenous random variable $Y$ with fixed probability $\mathbb{P}(Y = \mathscr{Y}) = \alpha$ and choose some point $\mathbf{x}_0$ for the observable data of interest. Then form the interval:
$$\text{CI}(1-\alpha) = \begin{cases}
[\theta_0] & & & \text{if } \mathbf{x} = \mathbf{x}_0 \text{ or } Y \in \mathscr{Y}, \\[6pt]
\Theta & & & \text{if } \mathbf{x} \neq \mathbf{x}_0 \text{ and } Y \notin \mathscr{Y}. \\[6pt]
\end{cases}$$
Assuming that $\mathbf{x}$ is continuous we have $\mathbb{P}(\mathbf{x} \neq \mathbf{x}_0) = 0$ and so the interval has the required coverage probability for all $\theta \in \Theta$. If $\mathbf{x} = \mathbf{x}_0$ then this interval is composed of a single point and so has length zero. This demonstrates that it is possible to formulate a confidence interval with length zero at an individual data outcome.
