# Different definitions of Bayes risk

I'm having trouble understanding the proper definition of Bayes risk. Let the data/variate $x \sim P(X|\theta)$, $\theta\in \Theta$, $\pi$ be a distribution on $\Theta$ (prior), $\hat \theta(x)$ be an estimator of $\theta$ based on a variates $x$, and $L$ a loss function between true and estimated parameters $\theta$. It appears to me that there are two different notions of Bayes risk around:

1. For given data , the Bayes risk is defined as $$\mathbb{E}[L(\theta, \hat\theta(x)]$$ with the expectation taken over the prior $\pi$, and $x$ fixed. This is what you'll find in Wikipedia, for instance.
2. In function estimation like wavelet theory, and when contrasting Bayes and minimax estimation, the risk is defined as $$R(\theta, \hat\theta) = \mathbb{E}[L(\theta, \hat\theta(x))]$$ where the expectation is taken over $P(X|\theta)$ (in regression, where $\theta$ is the signal, this means we integrate over the noise). For minimax estimation, we look at the maximum risk $$\max_{\theta\in\Theta} R(\theta, \hat\theta)$$ whereas for the Bayes risk, we put a prior $\pi$ in $\Theta$, so the Bayes risk is defined as $$\mathbb{E}[R(\theta, \hat\theta)]$$ with the expectation taken with respect to $\pi$.

So one is defined for a specific variate, using the loss function, whereas the other is defined for the expected loss (risk) across all variates. I was wondering if these are really two different things, or my understanding is off, or if they are two sides of the same thing. I would appreciate any clarification.

• There's actually a 3rd one. In ML/Statistical Learning Theory, the Bayes risk is defined as $\arg\max_f\mathbb{E}_{X,Y}[f(X) \not = Y]$, and the optimal $f^*$ is called the bayes classifier (see for eg. Devroye et al. 1996). Does anyone know why? Is it only by analogy to the case here (with $y$ replacing $\theta$)? Jan 19, 2020 at 5:03

To quote from my book, The Bayesian Choice (2007, Section 2.3, pp. 63-64)

The Bayesian approach to decision theory integrates on the parameter space $$\Theta$$, since $$\theta$$ is unknown, instead of integrating on the sampling space $${\cal X}$$, as $$x$$ is observed. It relies on the posterior expected loss $$\begin{eqnarray*} \rho(\pi,d|x) & = & \mathbb{E}^\pi[L(\theta,d)|x] \\ \tag{1} & = & \int_{\Theta} \mathrm{L}(\theta,d) \pi(\theta|x)\, \text{d}\theta, \end{eqnarray*}$$ which averages the error (i.e., the loss) according to the posterior distribution of the parameter $$\theta$$, conditional on the observed value $$x$$. Given $$x$$, the average error resulting from decision $$d$$ is actually $$\rho(\pi,d|x)$$. The posterior expected loss is thus a function of $$x$$ but this dependence is not an issue, as opposed to the frequentist dependence of the risk on the parameter because $$x$$, contrary to $$\theta$$, is known.

Given a prior distribution $$\pi$$, it is also possible to define the integrated risk, which is the frequentist risk averaged over the values of $$\theta$$ according to their prior distribution $$\begin{eqnarray*} r(\pi,\delta) & = & \mathbb{E}^\pi[R(\theta,\delta)] \\ \tag{2} & = & \int_{\Theta} \int_{\cal X} \mathrm{L}(\theta,\delta(x))\, f(x|\theta) \,\text{d}x\ \pi(\theta)\, \text{d}\theta. \end{eqnarray*}$$ One particular appeal of this second concept is that it associates a real number with every estimator, not a function of $$\theta$$. It therefore induces a total ordering on the set of estimator s, i.e., allows for the direct comparison of estimators. This implies that, while taking into account the prior information through the prior distribution, the Bayesian approach is sufficiently reductive (in a positive sense) to reach an effective decision. Moreover, the above two notions are equivalent in that they lead to the same decision.

A bit further, I use the following definition for the Bayes risk:

A Bayes estimator associated with a prior distribution $$\pi$$ and a loss function $$\mathrm{L}$$ is any estimator $$\delta^\pi$$ which minimizes $$r(\pi,\delta)$$. For every $$x\in\cal{X}$$, it is given by $$\delta^\pi(x)$$, argument of $$\min_d \rho(\pi,d|x)$$ The value $$r(\pi) = r(\pi,\delta^\pi)\tag{3}$$ is then called the Bayes risk.

• Thanks! So to clarify: Bayes risk in 1. = your posterior expected loss, risk in 2. = your frequentist risk, Bayes risk in 2. = your integrated risk, and the risk attained by the minimizing (Bayes) estimator for 2. = your Bayes risk, correct? Also, is it true that $\delta^\pi$ always minimizes both globally as well as for each $x$, as you seem to imply? Mar 27, 2018 at 15:23
• Yes, this is a correct summary. And the Bayes estimate is found by minimising the posterior expected risk for each value of $x$, provided Fubini's theorem applies. Mar 27, 2018 at 15:40
• @Josh: I am afraid I do not see the link with empirical risk in either question or answer. Aug 2, 2020 at 14:10
• Thanks @Xi'an I just fixed the comment: Re-posting: this is a great answer (BTW I love your book!). QQ on the above, where does Empirical risk fit above? I know it's a Monte Carlo approximation of a type of statistical risk, but which type?
– Josh
Aug 2, 2020 at 15:32