5
$\begingroup$

This thread asks, Are loss functions necessarily additive in observations? As of now, one answer is in the negative. However, I am not aware of any practical examples of nonadditive loss functions that are intuitively justifiable in real-world problems. Therefore, I am asking for a realistic/intuitive example where a nonadditive loss function is preferred over additive ones.

To distinguish between

  • loss functions used as objective functions in estimation and their additivity w.r.t. to the training data

and

  • loss functions used for evaluating predictions and their additivity w.r.t. to the test data,

the focus of this question is on the latter (assessing point predictions vs. actual realizations, for example).

$\endgroup$
  • $\begingroup$ How about General Least Squares or Feasible General Least Sqaures with $\Omega$ being a non-diagonal matrix $$(y - \mathbf Xb)^\top \Omega^{-1} (y - \mathbf Xb)$$ $\endgroup$ – Stop Closing Questions Fast Oct 22 '19 at 13:40
  • $\begingroup$ Thank you, Jesper. First, I suppose FGLS applies to estimation but not to prediction; I am specifically interested in the latter. Second, isn't FGLS additive in observations? $\endgroup$ – Richard Hardy Oct 22 '19 at 13:47
  • $\begingroup$ FGLS can give rise to other $\hat \beta$ parameters than $\hat \beta_{OLS}$ hence predicted values will also differ from OLS and in sample FGLS has efficiency properties that could be relevant for prediction see for example stats.stackexchange.com/questions/14426/prediction-with-gls. As for being a sum of observations ... it is a double sum $\sum_i \sum_j \omega_{ij} \hat \epsilon_i \hat \epsilon_j$ which when $\Omega^{-1}$ is non-diagonal involves cross-products of observations residuals so it is not a sum that can be written like $\sum_i f(\hat \epsilon_i)$. $\endgroup$ – Stop Closing Questions Fast Oct 22 '19 at 14:11
  • $\begingroup$ Otherwise you can look at GMM estimation for a general estimation framework where the function minimized is not a sum over observations but weighted double sum $m^\top Wm$ of sample moments. $\endgroup$ – Stop Closing Questions Fast Oct 22 '19 at 14:23
  • $\begingroup$ Thank you! $\hat\epsilon_t$ is not data ($y$ and $X$ are), but I guess the solution might be nonadditive in $y$. I also see I was unclear in my question formulation: I am interested in loss functions used to evaluate predictions, not used as objective functions in estimation. (I will update the post accordingly.) As such, I believe the example is (becomes) not relevant. $\endgroup$ – Richard Hardy Oct 22 '19 at 14:55
0
$\begingroup$

A sensible choice of loss could be the negative of utility. Within the framework of maximization of expected utility (MEU), we would have additive loss, since we would use average (over the set of test cases) negative loss as an estimate of expected utility. On the other hand, we could have nonadditive loss if we abandon MEU. E.g. if we are trying to maximize the utility of the worst outcome, we could choose the negative maximum (over the set of test cases) loss as an estimate of our target. A key observation here is that the maximum is not an additive function.

$\endgroup$
0
+50
$\begingroup$

One example that comes to mind is the area under the ROC curve (AUC). For binary classification problems where the model outputs a continuous score (e.g. logistic regression or SVMs), AUC gives the probability that the model will score a randomly selected 'positive' instance higher than a randomly selected 'negative' instance.

For evaluating prediction performance, AUC plays the same role as other metrics/loss functions (e.g. misclassification rate, log loss, etc). Namely, it maps predicted scores and true labels to a real number that summarizes performance. And, it can be used as the basis for decision rules; in particular, as an objective function for model selection. Higher AUC is more desirable, so AUC is actually a utility function rather than a loss function. But, this distinction is minor, as one can simply multiply AUC by negative one to obtain the loss incurred by choosing a particular model.

Unlike misclassification rate, log loss, etc., AUC is non-additive (in the sense defined in the question). That is, if $y_i$ and $s_i$ are the true label and predicted score for the $i$th test case and $g$ is an arbitrary function, AUC can't be expressed in the form $\sum_{i=1}^n g(y_i, s_i)$. Rather, AUC is calculated by integrating the estimated ROC curve, which consists of the true positive rate vs. false positive rate as the classification threshold is varied. The integral is typically calculated using the trapezoid rule between points on the ROC curve. Although this involves a sum over trapezoids, AUC is non-additive because the area of each trapezoid depends non-additively on the predicted score and true labels of multiple test cases. For details, see section 7 and algorithm 2 in Fawcett (2006).

Bradley (1997), Huang and Ling (2005), and others have argued for the use of AUC over accuracy (which is additive). Although AUC has found wide use (e.g. ~247k google scholar results for +auc +classification), there are arguments against it as well; e.g. see Lobo et al. (2008).

References

  • Fawcett, T. (2006). An introduction to ROC analysis. Pattern recognition letters, 27(8), 861-874.

  • Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7), 1145-1159.

  • Huang, J., & Ling, C. X. (2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on knowledge and Data Engineering, 17(3), 299-310.

  • Lobo, J. M., Jimenez‐Valverde, A., & Real, R. (2008). AUC: a misleading measure of the performance of predictive distribution models. Global ecology and Biogeography, 17(2), 145-151.

$\endgroup$
  • 1
    $\begingroup$ Very interesting. Can the AUC be considered a loss function, though? To me it looks more like a summary metric for a range of different loss functions (points on the curve)? Does AUC not start as a vector each coordinate of which is an additive function of the test data, and then the coordinates are just interpolated? $\endgroup$ – Richard Hardy Feb 11 at 7:37
  • $\begingroup$ @RichardHardy I tried to address your questions in an edit. Briefly: AUC can be considered a utility function (and flipping the sign gives a loss function). And, it is indeed non-additive. I'd still like the description to be a bit more clear, but will have to return to it later. $\endgroup$ – user20160 Feb 12 at 2:02
  • $\begingroup$ I will have to think more about whether I find your reasoning convincing. E.g. I believe people derive utility from outcomes, not from parameters or estimates, so I am not sure about your comment that AUC can be considered a utility function. In any case, the bounty is yours. It would have been a shame to waste it. $\endgroup$ – Richard Hardy Feb 12 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.