Show the shortest confidence interval of a normal distribution

Question

I'm having trouble formally showing a problem I have been given. It goes as so:

Show that among all $(1-\alpha)*100$% confidence intervals of the form $$\bar{X}-Z_{k\alpha}*\frac{\sigma}{\sqrt{n}}<\mu<\bar{X}+Z_{(1-k)\alpha}*\frac{\sigma}{\sqrt{n}}$$

the one with $k = 0.5$ is the shortest.

Now, I feel I understand this problem conceptually pretty well. For the normal distribution, and really any distribution that is symmetric about the mean and is strictly decreasing away from the mean, the further away you go from the mean the less area there is under the curve. If you want the shortest confidence intervals, then you want to maximize the area under the curve while minimizing the "horizonal" distance of the interval. Making $k=0.5$ makes the ends of both sides of the confidence interval $\frac{\alpha}{2}$, which are equally distant from the mean. If we change $k$, we must have the confidence interval include the same area, which will inevitably make the 'horizontal' distance greater since we have to go further out horizontally to include the same area.

I think this is the right idea but I have no idea how I am supposed to show it. I've tried setting up integrals and showing some sort of minimum interval with equal area but I haven't been very successful. Any help would be greatly appreciated!

Answer 1

The length of the interval is

$$\left(\bar{X}-Z_{(1-k)\alpha}\frac{\sigma}{\sqrt{n}}\right) - \left(\bar{X}-Z_{1 - k\alpha}\frac{\sigma}{\sqrt{n}}\right) = \left(Z_{1 - k\alpha} - Z_{(1-k)\alpha}\right)\frac{\sigma}{\sqrt{n}}. $$

Because when $k$ is varied $\sigma/\sqrt{n}$ remains constant, this is minimized provided $Z_{1 - k\alpha} - Z_{(1-k)\alpha}$ is minimized.

Another way to look at it is to write

$$z = Z_{(1-k)\alpha},\ w = Z_{1 - k\alpha}.$$

Because the interval $[z,w]$ should contain $1-\alpha$ probability and obviously both $z$ and $w$ will be finite at a minimum, necessarily

$$-\infty \lt z \lt Z_\alpha$$

and $k$ must lie between $0$ and $1$.

According to the Fundamental Theorem of Calculus, when $z$ is increased infinitesimally to $z+dz$, the probability of the interval decreases by $f(z)dz$ where $f$ is the PDF for $\bar X$. To compensate, $w$ must increase by an infinitesimal amount $dw$ for which

$$f(z)dz = f(w)dw.$$

In the figure, the interval $[z,w]$ has been shifted to $[z+dz,w+dw]$. To keep the probabilities the same, $dw$ is only about half of $dz$ because the height of the PDF at $z$, $f(z)$, is only about half the height at $w$. Therefore this shift has shrunk the interval. Shifting should continue until no more shrinking is possible, which will therefore occur when the heights at the interval endpoints are equal (as argued below).

At the same time the length of the interval, given by $w-z$, changes by $dw-dz$. A minimum will occur at a critical point, giving the criterion $0 = dw-dz$, implying by virtue of the preceding result that

$$f(z) = f(w).$$

For any unimodal continuous distribution with PDF $f$ there will be (practically by definition) at most two solutions to the equation $f(z) = c$ for any number $c$. Moreover, as $c$ decreases, those solutions--if they exist--must draw further apart. That shows there will be a single solution to the preceding equation, with $z$ less than the mode and $w$ greater than the mode, provided $0 \lt \alpha \lt 1/2$, and that it will be a global minimum. (For $\alpha=1/2$ the interval will reduce to a point. Although any point would do, a mode would be a point of greatest density. For $1/2\lt \alpha \lt 1$ there are no solutions.)

Finally, when the distribution is also symmetric (as in the case of a Normal distribution), then necessarily $z$ and $w$ must be equidistant from the mode, implying $k=1/2$.