# Loss function in Supervised Learning vs Statistical Decision Theory

I am confused by the different definitions of Loss Function in statistical decision theory vs machine learning.

In statistical decision theory, a loss function is typically defined as $$L(\theta, \delta(X))$$, where $$\theta$$ is the true, unknown parameter, $$\delta(.)$$ is the decision rule, and $$X$$ is data (generated from $$\theta$$?). See for example lectures of the Theory of Statistics class.

In machine learning, it seems the loss function is defined as $$L(y, f(X))$$, where $$y$$ is the true label and $$f(x)$$ is some model. See, for example, Elements of Statistical Learning chapter 2.4.

My question is if they are talking about the same thing. It seems different. For example, if I am going to predict the next coin toss of an unknown coin, I can then model the coin toss as following a Bernoulli distribution with an unknown parameter $$\theta$$.

Let $$X$$ be some historical data. Then it appears that the loss function from statistical decision theory is computing my prediction $$\delta(X)$$ against the unknown parameter $$\theta$$ whereas in ML, it is computing the same prediction $$\delta(X)$$ (or $$f(X)$$) against the true label?

I am having trouble reconciling the two concepts.

• A somewhat related question with essentially the same title: stats.stackexchange.com/questions/485964 Sep 4, 2021 at 18:05
• I checked out that post before I wrote up my question. Although the title seems similar, what is being asked is completely different. That post was talking about the difference in prediction time vs training time. I am talking about the difference in the forms of the loss functions themselves. Thanks for pointing it out. Sep 4, 2021 at 18:39
• It is helpful to consider the overarching goals of both statistical inference and supervised learning: to choose the estimator (or hypothesis) that minimizes a risk function (also known as the expected loss). Does this answer your question? stats.stackexchange.com/questions/540140/… Sep 4, 2021 at 20:34

In point prediction settings (such as a lot of ML), the decision is a potential value of the label, and the state of nature effectively simplifies to the true value of the label, so the loss $$L(y, \hat y)$$ can be written as the loss from predicting $$\hat y$$ when the truth is $$y$$.
In parametric inference settings, the decision is a potential value of the parameter, and the state of nature effectively simplifies to the true parameter value, so the loss $$L(\theta, \hat\theta)$$ can be written as the loss from estimating $$\hat\theta$$ when the truth is $$\theta$$.
• Thanks for the great answer. I still have two questions: 1) what if we can observe not the true state but a stochastic manifestation of it - e.g. a coin toss has an unobservable probability $\theta$ of turning head but we only observe the past history of the coin toss. It can still be formulated as a supervised learning problem (say, there are many coins and we can describe them with features $X$), and the loss function will still be L(head/tail, prediction)? Another example is to predict which team will win a sports match. Sep 5, 2021 at 16:14
• 2) When the true state can be observed, i.e. $\theta$ is the same as $y$, I wonder how some of the concepts of "risk" in statistical decision theory carry over. Most importantly, the concept of posterior expected loss defined as $E[L(\theta, \delta(X))|X]$ with the expectation taken over the posterior $P(\theta|X)$ seems meaningless, as now $\theta$ is an observed truth. The Bayes risk, $E[L(\theta, \delta(X))]$, with the expectation taken over the joint distribution of $(\theta, X)$ is another something I no longer feel I understand in the ML setting. Sep 5, 2021 at 16:19