In all of the contexts I've seen loss functions in statistics/machine learning so far, loss functions are additive in observations. i.e.: loss $Q_D$ of dataset $D$ is an additive aggregation of losses at observations $i\in D$: $Q_D(\beta)=\sum_{i\in D}Q_i(\beta)$. e.g. in the loss that is a simple sum of squared residuals: $Q_D=\sum_i(y_i-X_i\beta)^2$.

This seems sensible, but I am wondering: Are there contexts in statistics/machine learning in which it happens (or reasons in theory why one might want) that a loss function is used that is not additive (or even separable) in observations?

  • 2
    $\begingroup$ It is not about additivity, but about taking average of your losses between the samples in your dataset. For the sole purpose of minimizing it, we can drop the 1/n constant, so arithmetic mean becomes a sum. $\endgroup$ – Tim Jun 7 '19 at 9:37
  • $\begingroup$ @Tim, sure but now the question becomes, do we ever want a loss function that is NOT the average of some function for each separate observation. This just reframes the question, since every additively separable function can be seen as an "average" by multiplying by 1/n, and vice versa. The question remains the same. $\endgroup$ – user56834 Jun 7 '19 at 10:02
  • $\begingroup$ A related question is posted here: Realistic/intuitive example where a nonadditive loss function is preferred over additive ones. $\endgroup$ – Richard Hardy Oct 22 '19 at 9:04

Loss functions are not always additive in observations: A loss function is function of an estimator (or predictor) and the thing it is estimating (predicting). The loss function is often, but not always, a distance function. Moreover, the estimator (predictor) sometimes, but not always, involves a sum of terms involving a single observation. Generally speaking, the loss function does not always have a form that is additive with respect to the observations. For prediction problems, deviation from this form occurs because of the form of the loss function. For estimation problems, it occurs either because of the form of the loss function, or because of the form of the estimator appearing in the loss function.

To see the generality of the loss form for a prediction problem, consider the general case where we have an observed data $\mathbf{y} = (y_1,...,y_n)$ and we want to predict the observable vector $\mathbf{y}_* = (y_{n+1},...,y_{n+k})$ using the predictor $\hat{\mathbf{y}}_* = \mathbf{H}(\mathbf{y})$. We can write the loss for this prediction problem as:

$$L(\hat{\mathbf{y}}_*, \mathbf{y}_*) = L(\mathbf{H}(\mathbf{y}), \mathbf{y}_*).$$

The loss function in your question is the Euclidean distance between the prediction vector and the observed data vector, which is $L(\hat{\mathbf{y}}_*, \mathbf{y}_*) = ||\hat{\mathbf{y}}_* - \mathbf{y}_*||^2 = \sum_i (\hat{y}_{*i} - y_{*i})^2$. That particular form is composed of a sum of terms involving the observed values being predicted, and so the additivity property holds in that case. However, there are many other examples of loss functions that give rise to a form that does not have this additivity property.

A simple example of two loss functions that are not additive in the observations are when the loss is equal to the prediction error either from the best prediction, or from the worst prediction. In the case of "loss from best prediction" we have the loss function $L(\hat{\mathbf{y}}_*, \mathbf{y}_*) = \min_i |\hat{y}_{*i} - y_{*i}|$, and in "loss from worse prediction" we have the loss function $L(\hat{\mathbf{y}}_*, \mathbf{y}_*) = \max_i |\hat{y}_{*i} - y_{*i}|$. In either case, the loss function is not additive for the individual terms.

  • $\begingroup$ Could you perhaps add an example of a loss function that is not additive in observations? $\endgroup$ – Richard Hardy Sep 10 '19 at 10:25
  • $\begingroup$ Still pretty interested in an example... $\endgroup$ – Richard Hardy Oct 20 '19 at 10:21
  • $\begingroup$ @RichardHardy: I just edited to add a couple of examples. Hope that is useful. $\endgroup$ – Ben - Reinstate Monica Oct 21 '19 at 20:53
  • $\begingroup$ Thank you! I do not see a technical problem with your examples, but I wonder about applications in which these kinds of loss functions could be relevant. Could you describe a hypothetical situation where $\min$ or $\max$ loss would be a logical choice? I guess I could imagine how they could be relevant when predicting the same observation with multiple attempts, but I cannot come up with an example where this works for predicting different observations. $\endgroup$ – Richard Hardy Oct 22 '19 at 5:19
  • $\begingroup$ @RichardHardy: Yeah, I'm not really sure of the practical value of loss functions like this. They could arise in competition environments where there is a scoring rule like this --- e.g., you get judged solely on your best prediction, or your worst. It would be difficult to think of a practical example where it is not directly constructed. $\endgroup$ – Ben - Reinstate Monica Oct 22 '19 at 7:31

There are two most common causes for loss function being a sum/average.

First, you may simply define your loss as average of some metric. It's related to concept of risk minimization.

Second reason is that you use Maximum Likelihood or something related, like Maximum A Posteriori. Additivity comes from the fact that Maximum Likelihood solves

$$\arg\max_{\theta} P_{\theta}(Dataset) = \arg\max_{\theta} \prod_{x \in Dataset} P_{\theta}(x)$$

which is equal to $$\arg\min_{\theta} \sum_{x \in Dataset} -log(P_{\theta}(x)).$$

For example, if $P_{\theta}$ is Gaussian you'll get exactly mean squared error.

  • 1
    $\begingroup$ We formally do not need to use log-likelihoods, you can maximize the likelihood, so this is rather an example where we use product rather then a sum, isn't it? $\endgroup$ – Tim Jun 7 '19 at 10:08
  • $\begingroup$ I don't understand your reasoning. These optimization problems are (mathematically) perfectly equivalent, so it does as much sense to say that you optimize a sum as you optimize a product. Sums are used in practice because of better numeric properties (multiplying a lot of numbers close to zero can be problematic due to loss of precision) $\endgroup$ – Jakub Bartczuk Jun 7 '19 at 10:58
  • 1
    $\begingroup$ You call it a "reason" for using sum in cost function, but it is just another representation of it, not a "reason" for taking sum. $\endgroup$ – Tim Jun 7 '19 at 11:01
  • $\begingroup$ Ah, in this case you're right. I tried to explain where do sums come from in loss functions. Even if technically what is optimized is not necessarily a sum. $\endgroup$ – Jakub Bartczuk Jun 7 '19 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.