minimum value of $E[Y - g(X)]^2$ When is $E[Y - g(X)]^2$ at it's minimum value over possible transforms $g$?
I guess it's when $g(x) = E[Y|X]$. But i'm unable to prove it. Any ideas or hints how can i start the proof? (If i'm right about the answer)
 A: Add and substract $E(Y|X)$ in your expression and condition to $X$ to prove the undebraced expression is equal to zero.
\begin{align*}
 & E(Y - g(X))^2 = \\
&E (Y- E(Y|X))^2 + 2\underbrace{E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right]}_{=0} + E(E(Y|X) - g(X))^2  \\
&\geq E(Y- E(Y|X))^2
\end{align*}
To reply to the comments in the question here is why the underbraced expression is equal to zero.
$$E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right] = E \{ E\left[(Y - E(Y|X) (E(Y|X) - g(X))|X\right] \} =  E \{(E(Y|X) - g(X)) \underbrace{E \left[(Y - E(Y|X) |X\right]}_{=0} \} = 0 $$
Where the term $E(E(Y|X) - g(X))$ goes out the conditional expectation to $X$ because it is a function of $X$. 
A: $$\begin{align}
[Y-g(X)]^2 &= [Y-\mathbb{E}[Y \mid X]+\mathbb{E}[Y \mid X]-g(X)]^2 \\
&= [Y-\mathbb{E}[Y \mid X]]^2+[\mathbb{E}[Y \mid X]-g(X)]^2+2[Y-\mathbb{E}[Y \mid X]][\mathbb{E}[Y \mid X]-g(X)]
\end{align}$$
Now, by double expectation,
$$\mathbb{E}[Y-g(X)]^2 = \mathbb{E}[\mathbb{E}\left\{[Y-g(X)]^2 \mid X\right\}]\text{.}$$
Looking at the third term of $\mathbb{E}[Y - g(X)]^2$ above, notice that $$\mathbb{E}\left\{Y-\mathbb{E}[Y \mid X] \mid X\right\}= \mathbb{E}[Y \mid X] - \mathbb{E}[\mathbb{E}[Y \mid X]\mid X]=\mathbb{E}[Y \mid X] - \mathbb{E}[Y \mid X] = 0$$
so thus, we end up with 
$$\mathbb{E}[Y - g(X)]^2 = \mathbb{E}[Y-\mathbb{E}[Y \mid X]]^2+\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2\text{.}\tag{1}$$
Note that in $(1)$, the only term dependent on $g(X)$ on the right side is $\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2$. Since this is the expectation of a squared quantity, we know that 
$$[\mathbb{E}[Y \mid X]-g(X)]^2 \geq 0$$
which means that minimization happens when 
$$\mathbb{E}[Y \mid X]-g(X) = 0$$
or $$g(X) = \mathbb{E}[Y \mid X]\text{.}$$
A: Since answers have already been provided, here is a short intro to conditional expectations as $L^2$ projections, i.e. for an alternative path to conditional expectations in which the question asked is taken more or less as definition.
Intuitively, taking the expected value of $Y$ conditioned on $X$ means averaging $Y$ over events which produce the outcomes of $X$. If we think of the $\sigma$-algebra generated by $X$, $\sigma(X)$, as grouping events leading to the same outcomes of $X$, then we see that we want the conditional expectation $\mathbb{E}[Y|X]$ to be a r.v. over $\sigma(x)$, closest in some sense to the original $Y$.
Therefore, if we work in $L^2(\Omega, \mathcal{A}, P)$, with $\mathcal{A}$ a $\sigma$-algebra over $\Omega$, then the conditional expectation $\mathbb{E}[Y|X]$ has a natural definition as the $L^2$ projection of Y onto $L^2$ of $\sigma(X)$ since $L^2(\Omega, \sigma(X), P)$ is a closed subspace of $L^2(\Omega, \mathcal{A}, P)$:
$$\mathbb{E}[Y|X] = \operatorname{argmin}_g \mathbb{E}[(Y-g(X))^2]$$
over all Borel $g$ such that $g(X) \in L^2$ (recall that $Z$ is measurable wrt. $\sigma(X)$ iff there is some Borel $g$ such that $Z=g(X)$)
The case $L^1$ is not so straightforward because we lack the Hilbert space structure, see e.g. "Jacod, Protter: Probability essentials", Chapter 23.
Conditional expectations can be taken over any sub $\sigma$-algebra of $\mathcal{A}$. For a nice introduction starting from conditioning on $\sigma$-algebras and with pictures and intuition, see "Conditional Expectations without tears".
