(To see why I wrote this, check the comments below my answer to this question.)

Type III errors and statistical decision theory

Giving the right answer to the wrong question is sometimes called a Type III error. Statistical decision theory is a formalization of decision-making under uncertainty; it provides a conceptual framework that can help one avoid type III errors. The key element of the framework is called the loss function. It takes two arguments: the first is (the relevant subset of) the true state of the world (e.g., in parameter estimation problems, the true parameter value $\theta$); the second is an element in the set of possible actions (e.g., in parameter estimation problems, the estimate $\hat{\theta})$. The output models the loss associated with every possible action with respect to every possible true state of the world. For example, in parameter estimation problems, some well known loss functions are:

  • the absolute error loss $L(\theta, \hat{\theta}) = |\theta - \hat{\theta}|$
  • the squared error loss $L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2$
  • Hal Varian's LINEX loss $L(\theta, \hat{\theta}; k) = \exp(k(\theta - \hat{\theta})) - k(\theta - \hat{\theta}) - 1,\text{ } k \ne0$

Examining the answer to find the question

There's a case one might attempt to make that type III errors can be avoided by focusing on formulating a correct loss function and proceeding through the rest of the decision-theoretic approach (not detailed here). That's not my brief – after all, statisticians are well equipped with many techniques and methods that work well even though they are not derived from such an approach. But the end result, it seems to me, is that the vast majority of statisticians don't know and don't care about statistical decision theory, and I think they're missing out. To those statisticians, I would argue that reason they might find statistical decision theory valuable in terms of avoiding Type III error is because it provides a framework in which to ask of any proposed data analysis procedure: what loss function (if any) does the procedure cope with optimally? That is, in what decision-making situation, exactly, does it provide the best answer?

Posterior expected loss

From a Bayesian perspective, the loss function is all we need. We can pretty much skip the rest of decision theory -- almost by definition, the best thing to do is to minimize posterior expected loss, that is, find the action $a$ that minimizes $\tilde{L}(a) = \int_{\Theta}L(\theta, a)p(\theta|D)d\theta$.

(And as for non-Bayesian perspectives? Well, it is a theorem of frequentist decision theory -- specifically, Wald's Complete Class Theorem -- that the optimal action will always be to minimize Bayesian posterior expected loss with respect to some (possibly improper) prior. The difficulty with this result is that it is an existence theorem gives no guidance as to which prior to use. But it fruitfully restricts the class of procedures that we can "invert" to figure out exactly which question it is that we're answering. In particular, the first step in inverting any non-Bayesian procedure is to figure out which (if any) Bayesian procedure it replicates or approximates.)

Hey Cyan, you know this is a Q&A site, right?

Which brings me – finally – to a statistical question. In Bayesian statistics, when providing interval estimates for univariate parameters, two common credible interval procedures are the quantile-based credible interval and the highest posterior density credible interval. What are the loss functions behind these procedures?

  • $\begingroup$ Very nice. But are they the only loss functions justifying these procedures? $\endgroup$
    – guest
    Commented Mar 16, 2012 at 5:28
  • 1
    $\begingroup$ @Cyan>> Thanks for asking and answering the question for me :) I will read all this and upvote whenever possible. $\endgroup$ Commented Mar 16, 2012 at 10:53
  • 4
    $\begingroup$ Interesting quote from Berger's Statistical decision theory and Bayesian analysis: "we do not view credible sets as having a clear decision-theoretic role, and are therefore leery of 'optimality' approaches to selection of a credible set" $\endgroup$ Commented Mar 18, 2012 at 10:40
  • 1
    $\begingroup$ @Simon Byrne>> 1985 was a long time ago; I wonder if he still thinks that. $\endgroup$
    – Cyan
    Commented Mar 18, 2012 at 17:34
  • 1
    $\begingroup$ @Cyan: I don't know, but decision theory is the one part of Bayesian statistics that hasn't changed much over the past 27 years (there have been a few interesting results, but Berger's book is still the standard reference), especially when compared to the popularity minimax results in frequentist statistics. $\endgroup$ Commented Mar 19, 2012 at 11:20

2 Answers 2


In univariate interval estimation, the set of possible actions is the set of ordered pairs specifying the endpoints of the interval. Let an element of that set be represented by $(a, b),\text{ } a \le b$.

Highest posterior density intervals

Let the posterior density be $f(\theta)$. The highest posterior density intervals correspond to the loss function that penalizes an interval that fails to contain the true value and also penalizes intervals in proportion to their length:

$L_{HPD}(\theta, (a, b); k) = I(\theta \notin [a, b]) + k(b – a), \text{} 0 < k \le max_{\theta} f(\theta)$,

where $I(\cdot)$ is the indicator function. This gives the expected posterior loss

$\tilde{L}_{HPD}((a, b); k) = 1 - \Pr(a \le \theta \le b|D) + k(b – a)$.

Setting $\frac{\partial}{\partial a}\tilde{L}_{HPD} = \frac{\partial}{\partial b}\tilde{L}_{HPD} = 0$ yields the necessary condition for a local optimum in the interior of the parameter space: $f(a) = f(b) = k$ – exactly the rule for HPD intervals, as expected.

The form of $\tilde{L}_{HPD}((a, b); k)$ gives some insight into why HPD intervals are not invariant to a monotone increasing transformation $g(\theta)$ of the parameter. The $\theta$-space HPD interval transformed into $g(\theta)$ space is different from the $g(\theta)$-space HPD interval because the two intervals correspond to different loss functions: the $g(\theta)$-space HPD interval corresponds to a transformed length penalty $k(g(b) – g(a))$.

Quantile-based credible intervals

Consider point estimation with the loss function

$L_q(\theta, \hat{\theta};p) = p(\hat{\theta} - \theta)I(\theta < \hat{\theta}) + (1-p)(\theta - \hat{\theta})I(\theta \ge \hat{\theta}), \text{ } 0 \le p \le 1$.

The posterior expected loss is

$\tilde{L}_q(\hat{\theta};p)=p(\hat{\theta}-\text{E}(\theta|\theta < \hat{\theta}, D)) + (1 - p)(\text{E}(\theta | \theta \ge \hat{\theta}, D)-\hat{\theta})$.

Setting $\frac{d}{d\hat{\theta}}\tilde{L}_q=0$ yields the implicit equation

$\Pr(\theta < \hat{\theta}|D) = p$,

that is, the optimal $\hat{\theta}$ is the $(100p)$% quantile of the posterior distribution, as expected.

Thus to get quantile-based interval estimates, the loss function is

$L_{qCI}(\theta, (a,b); p_L, p_U) = L_q(\theta, a;p_L) + L_q(\theta, b;p_U)$.

  • 1
    $\begingroup$ Another way to motivate this is to re-write the loss function as a (weighted) sum of the width of the interval plus the distance, if any, by which the interval fails to cover the true $\theta$. $\endgroup$
    – guest
    Commented Mar 16, 2012 at 5:31
  • $\begingroup$ Is there any other way to think of quantile based intervals that doesn't directly reference quantiles or the length of the interval. I was hoping for something like "the quantile interval maximizes/minimizes the average/minimum/maximum/etc. something-measure" $\endgroup$ Commented Oct 8, 2014 at 9:57
  • $\begingroup$ @RasmusBååth, you're basically asking, "what are the necessary conditions on the loss function for quantile intervals to be the solution to the minimization of posterior expected loss?" My intuition, just from the way the math works in the forward direction, is that this is pretty much it. Haven't proven it, though. $\endgroup$
    – Cyan
    Commented Oct 9, 2014 at 14:53
  • $\begingroup$ So I'm not sure about a loss function, but I know of a procedure that, depending on the point loss function $L$, will result in either a HPD or a quantile interval. Assume you have random samples $s$ draw from the posterior. 1. Select the point in $s$ with the lowest posterior loss and add that point to your interval. 2. Remove that point from $s$, due to this removal the posterior loss for the remaining points in $s$ might now change (depending on $L$). 3. Be happy if your interval has the required coverage, otherwise repeat from (1). L = L0 gives HPD, L = L1 gives quantile interval. $\endgroup$ Commented Oct 12, 2014 at 22:11
  • 5
    $\begingroup$ just mentioning that Section 5.5.3 of Bayesian Choice covers the loss-based derivation of credible sets... $\endgroup$
    – Xi'an
    Commented Nov 21, 2014 at 13:09

Intervals of minimal size

One obvious choice of a loss function for interval selection (both Bayesian and frequentist) is to use the size of the intervals as measured in terms of the marginal distributions. Thus, start with the desired property or the loss function, and derive the intervals that are optimal. This tends not to be done, as is exemplified by the present question, even though it is possible. For Bayesian credible sets, this corresponds to minimize the prior probability of the interval, or to maximize the relative belief, e.g., as outlined in Evans (2016). The size may also be used to select frequentist confidence sets (Schafer 2009). The two approaches are related and can be implemented fairly easily via decision rules that preferentially included decisions with large pointwise mutual information (Bartels 2017).

Bartels, C.,2017. Using prior knowledge in frequentist tests. figshare. https://doi.org/10.6084/m9.figshare.4819597.v3

Evans, M., 2016. Measuring statistical evidence using relative belief. Computational and structural biotechnology journal, 14, pp.91-96.

Schafer, C.M. and Stark, P.B., 2009. Constructing confidence regions of optimal expected size. Journal of the American Statistical Association, 104(487), pp.1080-1089.

  • $\begingroup$ I see you're citing Evans per Keith O'Rourke's suggestion (andrewgelman.com/2016/07/17/…). I really like Evans's stuff. $\endgroup$
    – Cyan
    Commented Apr 14, 2017 at 19:07
  • $\begingroup$ I'm very pleased having been informed by Keith on work that starts differently but ends up at similar conclusions! Important to cite this. $\endgroup$
    – user36160
    Commented Apr 15, 2017 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.