I am reading Bishop's pattern recognition book. In the decision theory part he first derives that using a quadratic loss function implies that our estimate $y(x)$ should be the conditional expectation $\mathbb{E}[t|x]$. Then, armed with this knowledge he wants to show a simpler proof: \begin{align*}L(t, y(x)) &= (y(x) - t)^2 \\&= (y(x) - \mathbb{E}[t|x] + \mathbb{E}[t|x] - t)^2 \\&= (y - \mathbb{E}[t|x])^2 + 2(y(x) - \mathbb{E}[t|x])(\mathbb{E}[t|x] - t) + (\mathbb{E}[t|x] - t)^2\end{align*}

So far so good but now he says "performing the integral over t, we see that the cross-term vanishes and we obtain an expression for the loss function in the form"

$$\mathbb{E}[L] = \int (y - \mathbb{E}[t|x])^2 p(x) dx + \int (\mathbb{E}[t|x] - t)^2 p(x) dx$$

Now, how does the cross-term vanish exactly? Also how can the above expression still contain $t$ (he just integrated over it, so he probably means $(\mathbb{E}[t|x] - \mathbb{E}[t])^2$ ?).

This is on page 47 of Bishop: Pattern Recognition And Machine Learning


First, taking the expectation in both $t$ and $x$ leads to \begin{align*}\overbrace{\mathbb{E}[L(t, y(x))]}^\text{both $t$ and $x$ are rvs} &= \mathbb{E}[(y(x) - t)^2] \\&= \mathbb{E}[(y(x) - \mathbb{E}[t|x] + \mathbb{E}[t|x] - t)^2] \\&= \mathbb{E}[(y - \mathbb{E}[t|x])^2] + \mathbb{E}[2(y(x) - \mathbb{E}[t|x])(\mathbb{E}[t|x] - t)] + \mathbb{E}[(\mathbb{E}[t|x] - t)^2]\\ &= \mathbb{E}[(y - \mathbb{E}[t|x])^2] + 2\overbrace{\mathbb{E}[\underbrace{(y(x) - \mathbb{E}[t|x])}_\text{function of $x$}\underbrace{\mathbb{E}[(\mathbb{E}[t|x] - t)|x]}_\text{expectation over $t$ given $x$}]}^\text{expectation over $x$} + \mathbb{E}[(\mathbb{E}[t|x] - t)^2]\\&= \mathbb{E}[(y - \mathbb{E}[t|x])^2] + 2\mathbb{E}[(y(x) - \mathbb{E}[t|x])\underbrace{(\mathbb{E}[t|x]- \mathbb{E}[t|x])}_{\mathbb{E}[\mathbb{E}[t|x]|x]= \mathbb{E}[t|x]}] + \underbrace{\mathbb{E}[\mathbb{E}[(\mathbb{E}[t|x] - t)^2|x]]}_\text{double expectation}\\&= \mathbb{E}[(y - \mathbb{E}[t|x])^2] + 2\mathbb{E}[(y(x) - \mathbb{E}[t|x])\times 0] + \mathbb{E}[\mathrm{var}[t|x]]\\&= \underbrace{\mathbb{E}[(y - \mathbb{E}[t|x])^2]}_\text{squared bias} + \underbrace{\mathbb{E}[\mathrm{var}[t|x]]}_\text{independent of $y$}\end{align*} [where the double expectation on line 4 means first integrating according to $p(t|x)$ and then according to $p(x)$]. Hence, taking $y=\mathbb{E}[t|x]$ minimises the above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.