Loss functions in statistical decision theory vs. machine learning?

Question

I'm quite familiar with loss functions in machine learning, but am struggling to connect them to loss functions in statistical decision theory [1].

In machine learning, a loss function is usually only considered at training time. It's a differentiable function of two variables, loss(true value, predicted value), that you iteratively minimize over the training set to converge to (locally) optimal model weights.

In statistical decision theory, a loss function seems to be relevant at prediction time (?). You want to rationally choose a value for an unknown quantity, based on your assessment of its probable values, and your loss of making a wrong prediction.

What is the intuition of how these two concepts relate to each other?

[1] For example, in Ch 6.3 of "Machine Learning: A Probabilistic Approach" or Ch 2.4 of "Elements of Statistical Learning".

Answer 1

The loss that is of ultimate interest is the prediction loss (or decision loss). It represents real (financial/material/...) consequences of any given decision for the decision maker. It is this and only this loss that we want to minimize for its own sake rather than as an intermediate goal.

The training loss is an intermediate tool for building prediction models. It does not affect the welfare of the decision maker directly; its effects manifest themselves only via the prediction loss.

It may or may not be a good idea to match the training loss to the prediction loss.

• For example, suppose you have a sample generated by a Normal random variable. You have to predict a new observation from the same population, and your prediction loss is quadratic. Absent additional information, your best guess is the mean of the random variable. The best* estimate of it is the sample mean. It so happens that the type of training loss that is minimized by the sample mean is quadratic. Thus, here the training loss coincides with the prediction loss.
• Now suppose the situation is the same but your prediction loss is the absolute value of the prediction error. Absent additional information, your best guess is the median of the random variable. The best* estimate of it is the sample mean, not the sample median (because our sample is generated by a Normal random variable). As we already know, the training loss that yields the sample mean when minimized is quadratic. Thus, here the training loss does not coincide with the prediction loss.

*Best in the sense of minimizing the expected prediction loss.

Answer 2

Let me give a slightly more ML-focused perspective on the accepted answer.

Don't conflate training loss and decision loss – they're separate concepts even though the functions can be the same. It's easier to see this distinction in classification than regression.

So, let's say we're doing binary classification using logistic regression. The training loss is cross entropy / log loss (maybe with regularization). After the model is trained, we don't care about the training loss anymore.

At prediction time, our logistic regression model tells us $P(y|x)$. We need to translate this distribution into a single class. Do we just pick the class with the highest probability? Do we want to be extra careful about false positives? We formally encode these preferences into a decision loss, which allows us to optimally choose a single class from $P(y|x)$.

---

For a more academic exposition, I found "Pattern Recognition and Machine Learning" to have a great disambiguation of these two.

Determination of $p(x, t)$ from a set of training data is an example of inference and is typically a very difficult problem whose solution forms the subject of much of this book. In a practical application, however, we must often make a specific prediction for the value of $t$, and this aspect is the subject of decision theory.... We shall see that the decision stage is generally very simple, even trivial, once we have solved the inference problem.

It is worth distinguishing between the squared loss function arising from decision theory and the sum-of-squares error function that arose in the maximum likelihood estimation of model parameters. We might use more sophisticated techniques than least squares, for example regularization or a fully Bayesian approach, to determine the conditional distribution $p(t|x)$. These can all be combined with the squared loss function for the purpose of making predictions.

Answer 3

Actually this is not really a big difference between Statistics and Machine Learning. Machine Learning theory is concerned with how well predictions work outside the training sample in terms of the loss function as well. I think this is usually referred to as generalization risk or generalization error there, see for example Bousquet & Elisseef: Stability and Generalization.

Obviously if you only have the training sample, you can only evaluate the loss function on the training data. But many methods are based on some kind of training loss minimization, which implies that the training error (because it is optimized on the training data) will not generalize well and loss on new observations can be expected to be higher. This depends on the specific method and situation, but considering at least theoretically (or on separate test data) applying the loss function to new to be predicted data is a key tool for investigating this, and both Statistics and Machine Learning are concerned with this. (And you can sometimes choose methods or parameters based on expected generalization loss rather than plain training loss, at least where theory exists.)

Answer 4

1. Theoretical approach: In first layer, define prediction error $\epsilon=y-E(y|x)$. Then, to estimate parameter, we define a loss or criterion function as $l(\epsilon)=\|\epsilon\|_p^p$.
2. Empricial approach: We define residual error $r=y-\hat{y}$, where $\hat{y}$ approximate the prediction function $E(y|x)$ from the given data under a suitable model structure with a suitable loss or criterion function $l(r)=\|r\|_p^p$.