# What is the difference between the risk function used in Bayesian inference and the one used in supervised learning?

In the context of Bayesian inference, given

• the random parameter $$\Theta$$,
• the observed data $$\mathcal{D} = \{x_1,x_2,\dots,x_N\}$$,
• the posterior $$p(\theta\mid \mathcal{D})$$,
• the estimator $$\hat\theta(\mathcal{D})$$,
• and the loss function $$L\left(\hat\theta(\mathcal{D}),\Theta\right)$$,

the Bayesian risk function is defined as $$R_B\left(\hat\theta,\mathcal{D}\right) = \mathbb{E}_{p(\theta\mid\mathcal{D})}\left[L\left(\hat\theta(\mathcal{D}),\Theta\right) \right] = \int_\theta L\left(\hat\theta(\mathcal{D}),\theta\right) \cdot p(\theta\mid\mathcal{D}) \ \text{d}\theta$$ In contrast, in the context of supervised learning, given

• the joint distribution $$p(x,y)$$,
• the hypothesis $$h(x)$$,
• and the loss function $$L\left(h(x),y\right)$$,

the supervised learning risk function is defined as $$R_{SL}\left(h\right) = \mathbb{E}_{p(x,y)}\left[L\left(h(X),Y\right) \right] = \int_x \int_y L\left(h(x),y\right) \cdot p(x,y) \ \text{d}x \ \text{d}y$$

Is there a relationship between $$R_B\left(\hat\theta,\mathcal{D}\right)$$ and $$R_{SL}\left(h\right)$$?

If we let $$Y := \Theta \\ h := \hat\theta$$ then $$R_{SL}(h)$$ becomes $$R_{SL}(\hat\theta) = \mathbb{E}_{p(x,\theta)}\left[L\left(\hat\theta(X),\Theta\right) \right]$$ Using the law of total expectation, \begin{align} R_{SL}(\hat\theta) &= \mathbb{E}_{p(x,\theta)}\left[L\left(\hat\theta(X),\Theta\right) \right]\\ &= \mathbb{E}_{p(x)}\left[\mathbb{E}_{p(\theta\mid x)}\left[L\left(\hat\theta(X),\Theta\right) \right] \right] \end{align} Without loss of generality, if we let $$X := (X_1,X_2,\dots,X_N)$$ then \begin{align} R_{SL}(\hat\theta) &= \mathbb{E}_{p(x)}\left[\mathbb{E}_{p(\theta\mid x)}\left[L\left(\hat\theta(X),\Theta\right) \right] \right] \\ &= \mathbb{E}_{p(\mathcal{D})}\left[\mathbb{E}_{p(\theta\mid \mathcal{D})}\left[L\left(\hat\theta(\mathcal{D}),\Theta\right) \right] \right] \end{align} Note that $$\mathbb{E}_{p(\theta\mid \mathcal{D})}\left[L\left(\hat\theta(\mathcal{D}),\Theta\right) \right] = R_B(\hat\theta,\mathcal{D})$$ and so $$R_{SL}(\hat\theta) = \mathbb{E}_{p(\mathcal{D})}\left[R_B(\hat\theta,\mathcal{D})\right]$$ This means that $$R_{SL}(\hat\theta)$$ is just the Bayesian risk averaged over all possible datasets $$\mathcal{D}$$.