Chi-squared confidence interval for variance When constructing, for example, a $90\%$ confidence interval for the population variance using the chi-squared distribution, we have:
\begin{align}
& P\left(a<\frac{(n-1)S^2}{\sigma^2}<b\right) \\
= {} & P\left(\frac{\sum(\bar{X}-X_i)^2}{b}<\sigma^2<\frac{\sum(\bar{X}-X_i)^2}{a}\right)=0.9, \\ & \text{ where } \frac{(n-1)S^2}{\sigma^2}\sim\chi_{n-1}^2.
\end{align}
In my course, we then find $a, b$ such that $$P(\chi_{n-1}^{2}<a)=P(\chi_{n-1}^{2}>b)=0.05.$$
My question is, given that the chi-squared distribution is asymmetric for small $n,$ why do we pick $0.05$ for both sides? Surely we’d have a shorter confidence interval if we weighted more of the $0.1$ probability to one of the sides, instead of splitting equally?
 A: For univariate continuous asymmetric distributions the highest density region (HDR) can be found by solving a constrained optimisation problem for the boundary points.  You are correct that this involves placing non-equal weight in the tails.  You can find a detailed analysis of this problem in O'Neill (2021), including a statement of the optimisation problem at issue and its solution.  This paper also goes through the problem of finding the optimal confidence interval for the variance.
To save you from reinventing the wheel, it is worth noting that HDRs for all standard univariate distributions are available in the stat.extend package in R.  The available families include the chi-squared distribution, the gamma distribution, and the inverse gamma distribution.  These can be used to manually compute the optimal confidence interval.  Alternatively, there are also direct functions for optimal confidence intervals, including the optimal confidence interval for the variance.  In the code below we use the CONF.var function to compute the optimal 95% confidence interval for some mock data.
#Load library
library(stat.extend)

#Create some mock data (same data as used by BruceET)
set.seed(2022)
x = rnorm(20, 50, 5)

#Compute optimal confidence interval
#Assumes a mesokurtic distribution (kurt = 3)
CONF.var(x, alpha = 0.05, kurt = 3)

        Confidence Interval (CI) 
 
95.00% CI for variance parameter for infinite population 
Interval uses 20 data points from data x with sample variance = 25.0148 and 
assumed kurtosis = 3.0000 
Computed using nlm optimisation with 8 iterations (code = 1) 

[12.4006846357447, 48.0126609150707]

A: $$
\Pr(a<\chi^2_k<b) = 0.9 \tag 1
$$
One way to choose $a$ and $b$ is to choose them so that the values of the chi-square density function at those two points are equal to each other, and at the same time so that line $(1)$ above is true.
That does give you a shorter confidence interval, but it is numerically somewhat complicated to implement, and probably the purpose in the course you're taking is just to show that it is possible to get a confidence interval for $\sigma^2$ by "inverting" the "pivotal quantity" $(n-1)S^2/\sigma^2.$
A: One answer is why not use equal probabilities for both tails? Suppose we do not, we then produce asymmetric confidence intervals in terms of probability. Let us take an extreme example, to make a 95% confidence interval, we would usually set the interval at 2.5% lower, and 2.5% upper tail. Now we decide we want an extreme example, so we choose 0% left tail and a 5% upper-tail of a heavy right tailed distribution. What is that? It is a 5% one-tailed (right) answer to the question of what the probability is of an $X_i\geq$ the larger confidence interval upper bound. So, why balance the probabilities? So that the probability of an answer lower than the confidence interval is the same the probability of an answer higher than the confidence interval, which then gives us a balanced or two-tailed answer, which is the probability of a $X_i\neq$ a value within the confidence interval.
It is not unreasonable to ask if there are situations in which one wants unequal probabilities for the tails. Here is an arbitrary example for which the probabilities are not the final answers sought. Suppose that the x-axis measure is a length of pipe that we have produced by cutting very long pipes in a factory, and let us further suppose that if a pipe is too short (i.e., less than our hypothetical confidence interval lower bound) we must discard it and that costs twice as much as a pipe that is too long, which too long pipe can be sent back for trimming to length. In that case, we might want a left tail that is half as probable as a right tail in order to balance the cost of having pipes of the wrong length.
