Suppose the 95% confidence interval for $\ln(x)$ is $[l,u]$. Is it true that the 95% CI for $x$ is simply $[e^l, e^u]$?
I have the intuition the answer is yes, because $\ln$ is a continuous function. Is there some theorem that supports/refutes my intuition?
That is a 95% confidence interval for $x$, but not the 95% confidence interval. For any continuous strictly-monotonic transformation, your method is a legitimate way to get a confidence interval for the transformed value. (For monotonically decreasing functions, you reverse the bounds.) The other excellent answer by tchakravarty shows that the quantiles match up for these transformations, which shows how you can prove this result.
Generally speaking, there are an infinite number of possible 95% confidence intervals you could formulate for $x$, and while this is one of them, it is not generally the shortest possible interval with this level of confidence. When formulating a confidence interval, it is usually best to try to optimise to produce the shortest possible interval with the required level of coverage --- that ensures that you can make the most accurate inference possible at the required confidence level. You can find an explanation of how to do this in a related question here.
Taking a nonlinear transformation of an existing interval does not give you the optimum (shortest) confidence interval (unless by an incredible coincidence!). The general method used to obtain the shortest confidence interval is to go back and look at the initial probability statement operating on the pivotal quantity used to formulate the interval. Instead of using "equal tails" in the probability statement, you set the relative tail sizes as a control variable, and then you find the formula for the length of the confidence interval conditional on that variable. Finally, you use calculus methods to determine the value of the control variable that minimises the interval length. Often this method can be programmed for broad classes of problems, allowing you to rapidly compute optimal confidence intervals for an object of interest.
You can easily show that this is the case.
Let $Y\equiv \log(X)$. Then, the $\alpha$-quantile of $Y$ is $y\in\mathbb{R}$, such that $\mathbb{P}[Y \leq y] = \alpha$. Similarly, the $\alpha$-quantile of $X$ is $x \in \mathbb{R}^+$, such that $\mathbb{P}[X \leq x] = \alpha$, or, $\mathbb{P}[\log(X) \leq y] = \alpha$, or, $\mathbb{P}[X \leq \exp(y)] = \alpha$. Thus, $y = \exp(x)$. Note that there are regularity conditions relating to the continuity and monotonicity of the transformation function $\log$ that you need to be careful about when applying this result more generally.
Confidence intervals are based on probabilities conditional on the parameters, and do not transform if you transform the parameters. Unlike (Bayesian) probabilities of the parameters (on which credible intervals are based). See for instance in this question: If a credible interval has a flat prior, is a 95% confidence interval equal to a 95% credible interval? a confidence interval is not just like a credible interval with a flat prior. For a confidence interval we have:
- The boundaries of probabilities (credibility intervals) will be different when you transform the variable, (for likelihood functions this is not the case). E.g for some parameter $a$ and a monotonic transformation $f(a)$ (e.g. logarithm) you get the equivalent likelihood intervals $$\begin{array}{ccccc} a_{\min} &<& a &<& a_{\max}\\ f(a_{\min}) &<& f(a) &<& f(a_{\max}) \end{array}$$
See in this question Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?
- You might see the confidence intervals as being constructed as a range of values for which an $\alpha$ level hypothesis test would succeed and outside the range an $\alpha$ level hypothesis test would fail.
That is, we choose the range of $\theta$ (as a function of $X$) based on a probability conditional on the $\theta$'s. For instance
$$I_{\alpha}(X) = \lbrace \theta: F_X(\alpha/2,\theta) \leq X \leq F_X(1-\alpha/2,\theta) \rbrace$$
the range of all hypotheses $\theta$ for which the observation is inside a two-tailed $\alpha\%$ hypothesis test.
This condition, the hypotheses, does not change with the transformation. For instance, the hypothesis $\theta = 1$, is the same as the hypothesis $\log(\theta) = 0$.
You could consider a 2d view of hypotheses on the x-axis and observations on the y-axis (see also The basic logic of constructing a confidence interval):
You could define a $\alpha$-% confidence region in two ways:
in vertical direction $L(\theta) < X < U(\theta)$ the probability for the data $X$, conditional on the parameter being truly $\theta$, to fall inside these bounds is $\alpha$ .
in horizontal direction $L(X) < \theta < U(X)$ the probability that an experiment will have the true parameter inside the confidence interval is $\alpha$%.
For the actual computation of the confidence interval we often use the vertical direction. We compute the boundaries for each $\theta$ as a hypothesis test. This computation will be the same for a transformed $\theta$.
So when you transform the parameter, then the image will just look the same, and only the scale on the x-axis will change. For a transformation of a probability density this is not the same and the transformation is more than just a change of the scale.
Indeed like Ben has answered. There is not a single confidence interval, and there are many ways to choose the boundaries. However, whenever the decision is to make the confidence interval based on probabilities conditional on the parameters, then the transformation does not matter (like the before mentioned $I_{\alpha}(X) = \lbrace \theta: F_X(\alpha/2,\theta) \leq X \leq F_X(1-\alpha/2,\theta) \rbrace$).
I would disagree that there is a shortest possible interval.
Or at least this can not be defined in a unique way, or possibly it can be defined based on the conditional distribution of observations, but in that case transformation (of the conditional part) does not matter.
In that case (based on conditional distribution) you define the boundaries such that the vertical direction is smallest (e.g. how people often make the smallest decision boundaries for a hypothesis test). This is the most common way to determine the confidence interval. Optimizing the confidence interval such that you get the smallest interval in the vertical direction is independent from transformations of the parameter (you can see this as stretching/deforming the figure in horizontal direction, which does not change the distance between the boundaries in vertical direction).
Making the boundaries smallest in the horizontal direction is more difficult, because there is no good way to define/measure it (making the interval shorter for one observation requires making the interval larger for another, and one would need some way to weigh the different observations). It could be possible, maybe, if you use some prior for the distribution of $\theta$. In that case one could shift the choice of the boundaries (which still must be in the vertical direction to ensure 95% coverage, conditional on $\theta$, but they do not need to be optimal in vertical direction) in order to optimise some measure for the length of the interval. In that case, the transformation does indeed change the situation. But this way of constructing confidence intervals is not very typical.
