What is the decision-theoretic justification for Bayesian credible interval procedures?

Question

(To see why I wrote this, check the comments below my answer to this question.)

Type III errors and statistical decision theory

Giving the right answer to the wrong question is sometimes called a Type III error. Statistical decision theory is a formalization of decision-making under uncertainty; it provides a conceptual framework that can help one avoid type III errors. The key element of the framework is called the loss function. It takes two arguments: the first is (the relevant subset of) the true state of the world (e.g., in parameter estimation problems, the true parameter value $\theta$); the second is an element in the set of possible actions (e.g., in parameter estimation problems, the estimate $\hat{\theta})$. The output models the loss associated with every possible action with respect to every possible true state of the world. For example, in parameter estimation problems, some well known loss functions are:

• the absolute error loss $L(\theta, \hat{\theta}) = |\theta - \hat{\theta}|$
• the squared error loss $L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2$
• Hal Varian's LINEX loss $L(\theta, \hat{\theta}; k) = \exp(k(\theta - \hat{\theta})) - k(\theta - \hat{\theta}) - 1,\text{ } k \ne0$

Examining the answer to find the question

There's a case one might attempt to make that type III errors can be avoided by focusing on formulating a correct loss function and proceeding through the rest of the decision-theoretic approach (not detailed here). That's not my brief – after all, statisticians are well equipped with many techniques and methods that work well even though they are not derived from such an approach. But the end result, it seems to me, is that the vast majority of statisticians don't know and don't care about statistical decision theory, and I think they're missing out. To those statisticians, I would argue that reason they might find statistical decision theory valuable in terms of avoiding Type III error is because it provides a framework in which to ask of any proposed data analysis procedure: what loss function (if any) does the procedure cope with optimally? That is, in what decision-making situation, exactly, does it provide the best answer?

Posterior expected loss

From a Bayesian perspective, the loss function is all we need. We can pretty much skip the rest of decision theory -- almost by definition, the best thing to do is to minimize posterior expected loss, that is, find the action $a$ that minimizes $\tilde{L}(a) = \int_{\Theta}L(\theta, a)p(\theta|D)d\theta$.

(And as for non-Bayesian perspectives? Well, it is a theorem of frequentist decision theory -- specifically, Wald's Complete Class Theorem -- that the optimal action will always be to minimize Bayesian posterior expected loss with respect to some (possibly improper) prior. The difficulty with this result is that it is an existence theorem gives no guidance as to which prior to use. But it fruitfully restricts the class of procedures that we can "invert" to figure out exactly which question it is that we're answering. In particular, the first step in inverting any non-Bayesian procedure is to figure out which (if any) Bayesian procedure it replicates or approximates.)

Hey Cyan, you know this is a Q&A site, right?

Which brings me – finally – to a statistical question. In Bayesian statistics, when providing interval estimates for univariate parameters, two common credible interval procedures are the quantile-based credible interval and the highest posterior density credible interval. What are the loss functions behind these procedures?

Answer 1

In univariate interval estimation, the set of possible actions is the set of ordered pairs specifying the endpoints of the interval. Let an element of that set be represented by $(a, b),\text{ } a \le b$.

Highest posterior density intervals

Let the posterior density be $f(\theta)$. The highest posterior density intervals correspond to the loss function that penalizes an interval that fails to contain the true value and also penalizes intervals in proportion to their length:

$L_{HPD}(\theta, (a, b); k) = I(\theta \notin [a, b]) + k(b – a), \text{} 0 < k \le max_{\theta} f(\theta)$,

where $I(\cdot)$ is the indicator function. This gives the expected posterior loss

$\tilde{L}_{HPD}((a, b); k) = 1 - \Pr(a \le \theta \le b|D) + k(b – a)$.

Setting $\frac{\partial}{\partial a}\tilde{L}_{HPD} = \frac{\partial}{\partial b}\tilde{L}_{HPD} = 0$ yields the necessary condition for a local optimum in the interior of the parameter space: $f(a) = f(b) = k$ – exactly the rule for HPD intervals, as expected.

The form of $\tilde{L}_{HPD}((a, b); k)$ gives some insight into why HPD intervals are not invariant to a monotone increasing transformation $g(\theta)$ of the parameter. The $\theta$-space HPD interval transformed into $g(\theta)$ space is different from the $g(\theta)$-space HPD interval because the two intervals correspond to different loss functions: the $g(\theta)$-space HPD interval corresponds to a transformed length penalty $k(g(b) – g(a))$.

Quantile-based credible intervals

Consider point estimation with the loss function

$L_q(\theta, \hat{\theta};p) = p(\hat{\theta} - \theta)I(\theta < \hat{\theta}) + (1-p)(\theta - \hat{\theta})I(\theta \ge \hat{\theta}), \text{ } 0 \le p \le 1$.

The posterior expected loss is

$\tilde{L}_q(\hat{\theta};p)=p(\hat{\theta}-\text{E}(\theta|\theta < \hat{\theta}, D)) + (1 - p)(\text{E}(\theta | \theta \ge \hat{\theta}, D)-\hat{\theta})$.

Setting $\frac{d}{d\hat{\theta}}\tilde{L}_q=0$ yields the implicit equation

$\Pr(\theta < \hat{\theta}|D) = p$,

that is, the optimal $\hat{\theta}$ is the $(100p)$% quantile of the posterior distribution, as expected.

Thus to get quantile-based interval estimates, the loss function is

$L_{qCI}(\theta, (a,b); p_L, p_U) = L_q(\theta, a;p_L) + L_q(\theta, b;p_U)$.

Answer 2

Intervals of minimal size

One obvious choice of a loss function for interval selection (both Bayesian and frequentist) is to use the size of the intervals as measured in terms of the marginal distributions. Thus, start with the desired property or the loss function, and derive the intervals that are optimal. This tends not to be done, as is exemplified by the present question, even though it is possible. For Bayesian credible sets, this corresponds to minimize the prior probability of the interval, or to maximize the relative belief, e.g., as outlined in Evans (2016). The size may also be used to select frequentist confidence sets (Schafer 2009). The two approaches are related and can be implemented fairly easily via decision rules that preferentially included decisions with large pointwise mutual information (Bartels 2017).

Bartels, C.,2017. Using prior knowledge in frequentist tests. figshare. https://doi.org/10.6084/m9.figshare.4819597.v3

Evans, M., 2016. Measuring statistical evidence using relative belief. Computational and structural biotechnology journal, 14, pp.91-96.

Schafer, C.M. and Stark, P.B., 2009. Constructing confidence regions of optimal expected size. Journal of the American Statistical Association, 104(487), pp.1080-1089.