1
$\begingroup$

I am reading a "Calibrated Structured Prediction by Kuleshov and Liang" link.

Calibration and sharpness. Given a forecaster $F : X → [0, 1]$, define $T(x) = \mathbb{E}[y| F(x)]$ to be the true probability of $y = 1$ given a that $x$ received a forecast $F(x)$. We can use $T$ to decompose the $l_2$ prediction loss as follows: $$ \mathbb{E}[(y-F(x))^2] = \mathbb{E}[(y-T(x))^2] + \mathbb{E}[(T(x)-F(x))^2]=\text{var}[y]-\text{var}[T(x)]+ \mathbb{E}[(T(x)-F(x))^2] $$ The first equality follows because $y−T(x)$ has expectation $0$ conditioned on $F(x)$, and the second equality follows from the variance decomposition of $y$ onto $F(x)$.

Trying to prove the above. What I have got so far. $$ \begin{align} \mathbb{E}[(y-F(x))^2] & = \mathbb{E}[(y-T(x)+T(x)-F(x))^2] \\\\ & = \mathbb{E}[(y-T(x))^2]+ \mathbb{E}[(T(x)-F(x))^2]+ 2\mathbb{E}[(y-T(x))(T(x)-F(x))]\\ \end{align} $$

For the last term in above using law of total expectation, conditioning on $F(x)$:

$$ \begin{equation*} \mathbb{E}[(y-T(x))(T(x)-F(x))]=\mathbb{E}[\mathbb{E}[(y-T(x))(T(x)-F(x))\vert F(x)]] \end{equation*} $$

The second parenthesis is a function of $F(x)$ so can come out of conditioning. $$ \begin{align*} \mathbb{E}[(y-T(x))(T(x)-F(x))]&=\mathbb{E}[(T(x)-F(x))\mathbb{E}[(y-T(x))\vert F(x)]]\\\\ & = \mathbb{E}[(T(x)-F(x))(T(x)-T(x))]=0 \end{align*} $$ Which used the definition of $T(x)$.

So far we have: $$ \begin{equation} \mathbb{E}[(y-F(x))^2]=\mathbb{E}[(y-T(x))^2] + \mathbb{E}[(T(x)-F(x))^2] \end{equation} $$

However, I can not close the derivation by proving: $$ \begin{equation*} \mathbb{E}[(y-T(x))^2] =\text{var}[y]-\text{var}[T(x)] \end{equation*} $$

$\endgroup$

1 Answer 1

0
$\begingroup$

I think the key insight is that from the definition of conditional expectation it follows that $$ \mathbb{E} [ y \mathbb{E} (y | F(x)) ] = \mathbb{E}[ \mathbb{E}(Y | F(x))^2 ] $$ Hence, \begin{align*} \mathbb{E}[(y-T(x))^2 ] &= \mathbb{E}[y^2] - 2 \mathbb{E} [ y \mathbb{E} (y | F(x)) ] + \mathbb{E}[\mathbb{E}(Y | F(x))^2] \\ &= \mathbb{E}[y^2] - \mathbb{E}[\mathbb{E}(Y | F(x))^2] \\ &= \mathbb{E}[y^2] - (\mathbb{E}[y])^2 - \left\{ \mathbb{E}[\mathbb{E}(Y | F(x))^2] - (\mathbb{E}[y])^2 \right\} \\ &= \mathrm{var}[y] - \mathrm{var}[\mathbb{E}(y | F(x))] , \end{align*} where the last equality follows from the law of total expectation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.