9
$\begingroup$

Suppose you want to estimate a linear model: ($n$ observations of the response, and $p+1$ predictors) $$\mathbb{E}(y_i) = \beta_0 + \sum_{j=1}^p \beta_j x_{ij}$$

One way to do this is through the OLS solution, i.e. choose the coefficients so that the sum of square errors is minimum:

$$(\beta_0,\beta_1,\cdots,\beta_p)^T = \underset{\beta_0,\beta_1,\cdots,\beta_p}{\arg \min} \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2 $$

Alternatively, you could use another loss function, like the sum of the absolute deviations, so that:

$$(\beta_0,\beta_1,\cdots,\beta_p)^T = \underset{\beta_0,\beta_1,\cdots,\beta_p}{\arg \min} \sum_{i=1}^{n} \left| y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right| $$

Suppose you have found the parameters for the two models, and want to choose the model with the smallest value of the loss function. How can you compare the minimum values attained by the loss functions in general? (i.e. not just this specific case - we could also try other $L_p$ based loss functions) There seems to be a difference in the scale of the functions - one deals with squares while the other does not.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Well, the Markov assumptions let us show that the first equation is BLUE, or the best linear unbiased estimator. In this case "Best" is determined by having the smallest standard errors of all possible estimators. Is that not an adequate statistic? $\endgroup$
    – gregmacfarlane
    Commented Aug 19, 2014 at 19:35
  • 2
    $\begingroup$ note that $\|x\|_2 \leq \|x\|_1 \leq \sqrt{n}\|x\|_2$ and so $$ \underset{\beta_0,\beta_1,\cdots,\beta_p}{\arg \min} \sqrt{\sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2} \leq \underset{\beta_0,\beta_1,\cdots,\beta_p}{\arg \min} \sum_{i=1}^{n} \left| y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right| \leq \sqrt(n) \underset{\beta_0,\beta_1,\cdots,\beta_p}{\arg \min} \sqrt{\sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2}$$ $\endgroup$
    – Manuel
    Commented Aug 19, 2014 at 19:38
  • 1
    $\begingroup$ @gmacfarlane I want to define "best" here in terms of the minimum value of the loss function, instead of the standard error on the parameter estimates. (just out of curiosity) $\endgroup$
    – Comp_Warrior
    Commented Aug 19, 2014 at 19:42
  • 2
    $\begingroup$ I think you cannot compare the fits that come from 2 different loss functions, because they are answers to different questions. Once you decide that a given loss function is the appropriate one for your situation, the fit follows from that decision. You cannot fold it back to validate the choice of loss function without this becoming circular. If you have some other criterion that both loss functions can be understood to be encompassed by, you could use that, but you need to have defined that in advance. $\endgroup$
    – gung - Reinstate Monica
    Commented Aug 19, 2014 at 20:40
  • 1
    $\begingroup$ Note that if the model for the expectation was correct and the sample sizes were large enough that both estimates were effectively at the population value, then the question would effectively convert to "How do I compare a mean deviation with a standard deviation"? Under any given distributional assumption you could then compare their expected size, but of course in small samples the estimates themselves differ. $\endgroup$
    – Glen_b
    Commented Aug 19, 2014 at 21:17

1 Answer 1

Reset to default
4
$\begingroup$

(Converting my comment into an answer.)

I think you cannot compare the fits that come from different loss functions, because they are answers to different questions. Once you decide that a given loss function is the appropriate one for your situation, the fit follows from that decision. You cannot fold it back to validate the choice of loss function without this becoming circular. If you have some other criterion that both loss functions can be understood to be encompassed by, you could use that, but you need to have defined that in advance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.