2
$\begingroup$

Suppose I am building a predictor for $y = f_w(x) + noise$ using some framework with parameters $w$ (linear regression, neural networks, etc.) given a number of training examples $\{(x_i,y_i)\}$.

I recall reading that finding the best $w$ in the sense of minimising the quadratic loss: $L(y, f_w(x)) = \sum_i (y_i - f_w(x_i))^2$ has the interpretation that that the learnt function $f_w(x)$ is such that $f_w(x) = E[y|x]$, assuming $(x,y)$ are drawn from some distribution $p(x,y)$.

I also recall reading that finding the best $w$ in the sense of minimising the absolute value loss: $L(y, f_w(x)) = \sum_i |y_i - f_w(x_i)|$ has the interpretation that $f_w(x) = Conditional Median(y|x)$, assuming $(x,y)$ are drawn from some distribution $p(x,y)$. (Please correct me if I am wrong.)

I'd like to know under what assumptions are the above statements valid? Are there some assumptions on the noise distribution? Should $p(x,y)$ take a particular form? Could you cite some formal references on this specific subject? It would be great if you could link to papers discussing the interpretation and proofs of other loss functions such as the log-loss, margin loss, etc.

$\endgroup$

1 Answer 1

1
$\begingroup$

I'd like to know under what assumptions are the above statements valid?

Yes, they are essentially valid.

Are there some assumptions on the noise distribution? Should p(x,y) take a particular form?

The conditional mean should exist, that's more or less all that is required. Note, that some very widely used distributions don't have the mean, such as Cauchy.

Could you cite some formal references on this specific subject? It would be great if you could link to papers discussing the interpretation and proofs of other loss functions such as the log-loss, margin loss, etc.

You can find any number of references on this subject, e.g. Granger, Clive WJ. "Outline of forecast theory using generalized cost functions." Spanish Economic Review 1.2 (1999): 161-173. This is a very well written short paper. Granger has a few others on the same subject, you can look them up in Google Scholar. His writing style is exquisite in clarity.

These slides have a simple derivation of optimal forecasts under mean square error, see p.17.

$\endgroup$
3
  • $\begingroup$ Thank you. I searched some more and I found another reference: "Building Cost Functions Minimizing to Some Summary Statistics (2000) by Marco Saerens" which also talks about this problem in detail. $\endgroup$
    – Vimal
    Dec 24, 2014 at 5:02
  • $\begingroup$ Congratulates for being the first Unsung Hero in CV. :) $\endgroup$ Jan 5, 2015 at 13:10
  • 1
    $\begingroup$ Thanks. It's such a strange distinction that one couldn't get it if tried :) $\endgroup$
    – Aksakal
    Jan 5, 2015 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.