23
$\begingroup$

When evaluating an estimator, the two probably most common used criteria are the maximum risk and the Bayes risk. My question refers to the latter one:

The bayes risk under the prior $\pi$ is defined as follows:

$$B_{\pi} (\hat{\theta}) = \int R(\theta, \hat{\theta} ) \pi ( \theta ) d \theta $$

I don't quite get what the prior $\pi$ is doing and how I should interpret it. If I have a risk function $R(\theta, \hat{\theta} )$ and plot it, intuitively I would take its area as a criterion to judge how "strong" the risk is over all possible values of $\theta$. But involving the prior somehow destroys this intuition again, although it is close. Can someone help me how to interpret the prior?

$\endgroup$
1
  • 2
    $\begingroup$ I fail to see how intuitive plotting the risk function can be when considering several parameters: in that setting, the functions intersect and do not identify a "best" estimator. The Bayes risk returns one single number for each estimator and hence allows for a ranking of all estimators. $\endgroup$
    – Xi'an
    Jan 14, 2017 at 14:32

2 Answers 2

28
$\begingroup$

[Here is an excerpt from my own textbook, The Bayesian Choice (2007), that argues in favour of a decision-theoretic approach to Bayesian analysis, hence of using the Bayes risk.]

Except for the most trivial settings, it is generally impossible to uniformly minimize (in $d$) the loss function $\text{L}(\theta,d)$ when $\theta$ is unknown. In order to derive an effective comparison criterion from the loss function, the frequentist approach proposes to consider instead the average loss (or frequentist risk) \begin{eqnarray*} R(\theta,\delta) & = & \mathbb{E}_\theta \lbrack \text{L} (\theta ,\delta(x))\rbrack \\ & = & \int_{\cal X} \text{L}(\theta,\delta(x))f(x|\theta) \,\text dx , \end{eqnarray*} where $\delta(x)$ is the decision rule, i.e., the allocation of a decision to each outcome $x\sim f(x|\theta)$ from the random experiment.

The function $\delta$, from ${\mathcal X}$ in $\mathfrak{D}$, is usually called estimator (while the value $\delta(x)$ is called estimate of $\theta$). When there is no risk of confusion, we also denote the set of estimators by $\mathfrak{D}$.

The frequentist paradigm relies on this criterion to compare estimators and, if possible, to select the best estimator, the reasoning being that estimators are evaluated on their long-run performance for all possible values of the parameter $\theta$. Notice, however, that there are several difficulties associated with this approach.

  1. The error (loss) is averaged over the different values of $x$ proportionally to the density $f(x|\theta)$. Therefore, it seems that the observation $x$ is not taken into account any further. The risk criterion evaluates procedures on their long-run performance and not directly for the given observation, $x$. Such an evaluation may be satisfactory for the statistician, but it is not so appealing for a client, who wants optimal results for her data $x$, not that of another's!
  2. The frequentist analysis of the decision problem implicitly assumes that this problem will be met again and again, for the frequency evaluation to make sense. Indeed, $R(\theta,\delta)$ is approximately the average loss over i.i.d. repetitions of the same experiment, according to the Law of Large Numbers. However, on both philosophical and practical grounds, there is a lot of controversy over the very notion of repeatability of experiments (see Jeffreys (1961)). For one thing, if new observations come to the statistician, she should make use of them, and this could modify the way the experiment is conducted, as in, for instance, medical trials.
  3. For a procedure $\delta$, the risk $R(\theta, \delta)$ is a function of the parameter $\theta$. Therefore, the frequentist approach does not induce a total ordering on the set of procedures. It is generally impossible to compare decision procedures with this criterion, since two crossing risk functions prevent comparison between the corresponding estimators. At best, one may hope for a procedure $\delta_0$ that uniformly minimizes $R(\theta,\delta)$, but such cases rarely occur unless the space of decision procedures is restricted. Best procedures can only be obtained by restricting rather artificially the set of authorized procedures.

Example 2.4 - Consider $x_1$ and $x_2$, two observations from $$ P_{\theta}(x = \theta-1) = P_{\theta}(x = \theta+1) = 0.5, \qquad \theta\in\mathbb{R}. $$ The parameter of interest is $\theta$ (i.e., $\mathfrak{D} = \Theta$) and it is estimated by estimators $\delta$ under the loss $$ \text{L}(\theta,\delta) = 1-\mathbb{I}_{\theta}(\delta), $$ often called $0-1$ loss, which penalizes errors of estimation, whatever their magnitude, by $1$. Considering the particular estimator

$$ \delta_0(x_1,x_2) = {x_1+x_2 \over 2}, $$ its risk function is \begin{eqnarray*} R(\theta,\delta_0) & = & 1-P_{\theta}(\delta_0(x_1,x_2) = \theta) \\ & = & 1-P_{\theta}(x_1 \ne x_2) = 0.5. \end{eqnarray*} This computation shows that the estimator $\delta_0$ is correct half of the time. Actually, this estimator is always correct when $x_1\ne x_2$, and always wrong otherwise. Now, the estimator $\delta_1(x_1,x_2) = x_1+1$ also has a risk function equal to $0.5$, as does $\delta_2(x_1,x_2) = x_2-1$. Therefore, $\delta_0$, $\delta_1$ and $\delta_2$ cannot be ranked under the $0-1$ loss. $\blacktriangleright$

On the contrary, the Bayesian approach to Decision Theory integrates on the space $\Theta$ since $\theta$ is unknown, instead of integrating on the space ${\cal X}$ as $x$ is known. It relies on the posterior expected loss \begin{eqnarray*} \rho(\pi,d|x) & = & \mathbb{E}^\pi[L(\theta,d)|x] \\ & = & \int_{\Theta} \text{L}(\theta,d) \pi(\theta|x)\, \text d\theta, \end{eqnarray*} which averages the error (i.e., the loss) according to the posterior distribution of the parameter $\theta$, conditionally on the observed value} $x$. Given $x$, the average error resulting from decision $d$ is actually $\rho(\pi,d|x)$. The posterior expected loss is thus a function of $x$ but this dependence is not troublesome, as opposed to the frequentist dependence of the risk on the parameter because $x$, contrary to $\theta$, is known.

$\endgroup$
4
  • 8
    $\begingroup$ So you are Christian Robert. I have met George Casella. I think you have published book(s) with him that I am aware of.. $\endgroup$ Jan 14, 2017 at 16:28
  • 5
    $\begingroup$ +1 answers don't get much better than that - great book by the way $\endgroup$ Jun 26, 2018 at 12:09
  • $\begingroup$ With all due respect, I think there is some confusion on the terminology, in my Deep Learning course theta refers to the parameterization of the decision function (the estimator \delta(x)) and the data x are pairs of (x,y) and the bayes risk is the minimal achievable expected risk over all possible functions, but you seem to refer to theta as the estimand, the data as the input and the bayes risk as the expected risk over all decisions. Would it be possible to clarify this? $\endgroup$
    – Sheed
    Aug 17, 2023 at 10:24
  • 1
    $\begingroup$ Bayesian inference is not a component of deep learning, even though the later may borrow some Bayesian concepts, so it is not a surprise if terminology and symbols differ. However, if you carefully read the above, nowhere do I state that the Bayes risk is an expectation over all decision functions. It is the smallest integrated risk over all decision functions. $\endgroup$
    – Xi'an
    Aug 17, 2023 at 16:33
6
$\begingroup$

Quoting the classical Statistical Decision Theory by James O. Berger:

[...] We have already stated that decision rules will be evaluated in terms of their risk functions $R(\theta, \delta)$. [...] The problem, as pointed out earlier, is that different admissible decision rules will have risks which are better for different $\theta$'s. To the rescue comes the prior $\pi(\theta)$, which supposedly reflects which $\theta$'s are the "likely" ones to occur. It seems very reasonable to "weight" $R(\theta, \delta)$ by $\pi(\theta)$ and average.

Yes you can evaluate $R(\theta, \delta)$ for each $\theta$, but then you would implicitly assume that each possible value of $\theta$ is equally likely. In Bayesian scenario you pick prior $\pi(\theta)$ that reflects probabilities of observing different $\theta$'s and include such information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.