# Loss functions in statistical decision theory vs. machine learning?

I'm quite familiar with loss functions in machine learning, but am struggling to connect them to loss functions in statistical decision theory [1].

In machine learning, a loss function is usually only considered at training time. It's a differentiable function of two variables, loss(true value, predicted value), that you iteratively minimize over the training set to converge to (locally) optimal model weights.

In statistical decision theory, a loss function seems to be relevant at prediction time (?). You want to rationally choose a value for an unknown quantity, based on your assessment of its probable values, and your loss of making a wrong prediction.

What is the intuition of how these two concepts relate to each other?

[1] For example, in Ch 6.3 of "Machine Learning: A Probabilistic Approach" or Ch 2.4 of "Elements of Statistical Learning".

The loss that is of ultimate interest is the prediction loss (or decision loss). It represents real (financial/material/...) consequences of any given decision for the decision maker. It is this and only this loss that we want to minimize for its own sake rather than as an intermediate goal.

The training loss is an intermediate tool for building prediction models. It does not affect the welfare of the decision maker directly; its effects manifest themselves only via the prediction loss.

It may or may not be a good idea to match the training loss to the prediction loss.

• For example, suppose you have a sample generated by a Normal random variable. You have to predict a new observation from the same population, and your prediction loss is quadratic. Absent additional information, your best guess is the mean of the random variable. The best* estimate of it is the sample mean. It so happens that the type of training loss that is minimized by the sample mean is quadratic. Thus, here the training loss coincides with the prediction loss.
• Now suppose the situation is the same but your prediction loss is the absolute value of the prediction error. Absent additional information, your best guess is the median of the random variable. The best* estimate of it is the sample mean, not the sample median (because our sample is generated by a Normal random variable). As we already know, the training loss that yields the sample mean when minimized is quadratic. Thus, here the training loss does not coincide with the prediction loss.

*Best in the sense of minimizing the expected prediction loss.

• I'm not sure I follow your second bullet. If a variable is truly normal, the mean coincides with the median. Why then would the sample median be a worse estimator? Except for very small samples, the sample estimates of the mean and median of a normally distributed variable should be very close to each other, right? – Frans Rodenburg Sep 4 '20 at 11:17
• @FransRodenburg, try a simulation if you do not trust my point. You will discover that sample mean is a more efficient estimator of $\mu$ than the sample median, for any sample size (small or large). This is a well known example, I did not come up with it myself. – Richard Hardy Sep 4 '20 at 21:28
• It sounds perfectly logical, I just don't see why this would be the case when the median should coincide with the mean. – Frans Rodenburg Sep 4 '20 at 21:31
• @FransRodenburg, there are many ways of estimating the same parameter. E.g. you could take the first element of the sample as an estimate of the mean. Obviously (?), this is inefficient compared to taking the sample mean. Less obvious but no less true is the fact that the sample median is a less efficient estimator than the sample mean when the data is normal. I would encourage to try a simulation just to see for yourself (I have tried it myself) even though I believed in the result upfront. – Richard Hardy Sep 4 '20 at 21:38
• Thank you for taking the time to clarify, but perhaps I'll turn this into another question. Namely, I have no reason to assume the first element would coincide with the mean, but I do have reason to assume the median would coincide with the mean. I suppose the question would then be: If the mean and median coincide, and the sample mean is a more efficient estimator of the mean, is the sample mean also a more efficient estimator of the median? – Frans Rodenburg Sep 5 '20 at 3:38

Let me give a slightly more ML-focused perspective on the accepted answer.

Don't conflate training loss and decision loss – they're separate concepts even though the functions can be the same. It's easier to see this distinction in classification than regression.

So, let's say we're doing binary classification using logistic regression. The training loss is cross entropy / log loss (maybe with regularization). After the model is trained, we don't care about the training loss anymore.

At prediction time, our logistic regression model tells us $$P(y|x)$$. We need to translate this distribution into a single class. Do we just pick the class with the highest probability? Do we want to be extra careful about false positives? We formally encode these preferences into a decision loss, which allows us to optimally choose a single class from $$P(y|x)$$.

For a more academic exposition, I found "Pattern Recognition and Machine Learning" to have a great disambiguation of these two.

Determination of $$p(x, t)$$ from a set of training data is an example of inference and is typically a very difficult problem whose solution forms the subject of much of this book. In a practical application, however, we must often make a specific prediction for the value of $$t$$, and this aspect is the subject of decision theory.... We shall see that the decision stage is generally very simple, even trivial, once we have solved the inference problem.

It is worth distinguishing between the squared loss function arising from decision theory and the sum-of-squares error function that arose in the maximum likelihood estimation of model parameters. We might use more sophisticated techniques than least squares, for example regularization or a fully Bayesian approach, to determine the conditional distribution $$p(t|x)$$. These can all be combined with the squared loss function for the purpose of making predictions.

Actually this is not really a big difference between Statistics and Machine Learning. Machine Learning theory is concerned with how well predictions work outside the training sample in terms of the loss function as well. I think this is usually referred to as generalization risk or generalization error there, see for example Bousquet & Elisseef: Stability and Generalization.

Obviously if you only have the training sample, you can only evaluate the loss function on the training data. But many methods are based on some kind of training loss minimization, which implies that the training error (because it is optimized on the training data) will not generalize well and loss on new observations can be expected to be higher. This depends on the specific method and situation, but considering at least theoretically (or on separate test data) applying the loss function to new to be predicted data is a key tool for investigating this, and both Statistics and Machine Learning are concerned with this. (And you can sometimes choose methods or parameters based on expected generalization loss rather than plain training loss, at least where theory exists.)