Proof of optimality of mean squared loss Suppose I am building a predictor for $y = f_w(x) + noise$ using some framework with parameters $w$ (linear regression, neural networks, etc.) given a number of training examples $\{(x_i,y_i)\}$.
I recall reading that finding the best $w$ in the sense of minimising the quadratic loss: $L(y, f_w(x)) = \sum_i (y_i - f_w(x_i))^2$ has the interpretation that that the learnt function $f_w(x)$ is such that $f_w(x) = E[y|x]$, assuming $(x,y)$ are drawn from some distribution $p(x,y)$.
I also recall reading that finding the best $w$ in the sense of minimising the absolute value loss: $L(y, f_w(x)) = \sum_i |y_i - f_w(x_i)|$ has the interpretation that $f_w(x) = Conditional Median(y|x)$, assuming $(x,y)$ are drawn from some distribution $p(x,y)$.  (Please correct me if I am wrong.)
I'd like to know under what assumptions are the above statements valid?  Are there some assumptions on the noise distribution?  Should $p(x,y)$ take a particular form?  Could you cite some formal references on this specific subject?  It would be great if you could link to papers discussing the interpretation and proofs of other loss functions such as the log-loss, margin loss, etc.
 A: 
I'd like to know under what assumptions are the above statements
  valid? 

Yes, they are essentially valid.

Are there some assumptions on the noise distribution? 
  Should
  p(x,y) take a particular form?

The conditional mean should exist, that's more or less all that is required. Note, that some very widely used distributions don't have the mean, such as Cauchy.

Could you cite some formal references
  on this specific subject? It would be great if you could link to
  papers discussing the interpretation and proofs of other loss
  functions such as the log-loss, margin loss, etc.

You can find any number of references on this subject, e.g. Granger, Clive WJ. "Outline of forecast theory using generalized cost functions." Spanish Economic Review 1.2 (1999): 161-173. This is a very well written short paper. Granger has a few others on the same subject, you can look them up in Google Scholar. His writing style is exquisite in clarity.
These slides have a simple derivation of optimal forecasts under mean square error, see p.17.
