6
$\begingroup$

I am confused by the different definitions of Loss Function in statistical decision theory vs machine learning.

In statistical decision theory, a loss function is typically defined as $L(\theta, \delta(X))$, where $\theta$ is the true, unknown parameter, $\delta(.)$ is the decision rule, and $X$ is data (generated from $\theta$?). See for example lectures of the Theory of Statistics class.

In machine learning, it seems the loss function is defined as $L(y, f(X))$, where $y$ is the true label and $f(x)$ is some model. See, for example, Elements of Statistical Learning chapter 2.4.

My question is if they are talking about the same thing. It seems different. For example, if I am going to predict the next coin toss of an unknown coin, I can then model the coin toss as following a Bernoulli distribution with an unknown parameter $\theta$.

Let $X$ be some historical data. Then it appears that the loss function from statistical decision theory is computing my prediction $\delta(X)$ against the unknown parameter $\theta$ whereas in ML, it is computing the same prediction $\delta(X)$ (or $f(X)$) against the true label?

I am having trouble reconciling the two concepts.

$\endgroup$
3
  • $\begingroup$ A somewhat related question with essentially the same title: stats.stackexchange.com/questions/485964 $\endgroup$ Sep 4, 2021 at 18:05
  • $\begingroup$ I checked out that post before I wrote up my question. Although the title seems similar, what is being asked is completely different. That post was talking about the difference in prediction time vs training time. I am talking about the difference in the forms of the loss functions themselves. Thanks for pointing it out. $\endgroup$ Sep 4, 2021 at 18:39
  • 1
    $\begingroup$ It is helpful to consider the overarching goals of both statistical inference and supervised learning: to choose the estimator (or hypothesis) that minimizes a risk function (also known as the expected loss). Does this answer your question? stats.stackexchange.com/questions/540140/… $\endgroup$
    – mhdadk
    Sep 4, 2021 at 20:34

1 Answer 1

4
$\begingroup$

I would say this is more a difference in the form of the decision than the loss. The loss function in both cases is Loss(true state of nature, your decision), but it simplifies differently depending on the form of the decision

In point prediction settings (such as a lot of ML), the decision is a potential value of the label, and the state of nature effectively simplifies to the true value of the label, so the loss $L(y, \hat y)$ can be written as the loss from predicting $\hat y$ when the truth is $y$.

In parametric inference settings, the decision is a potential value of the parameter, and the state of nature effectively simplifies to the true parameter value, so the loss $L(\theta, \hat\theta)$ can be written as the loss from estimating $\hat\theta$ when the truth is $\theta$.

There are more complicated settings, too. For example, your decision might be an interval, and the state of nature might be a value, and the loss could be the length of the interval plus the distance from the value to the closest point of the interval (possibly zero)[PDF]. In that setting there isn't the nice correspondence between potential decisions and potential states of nature, and the loss doesn't simplify down to a summary of the error in the decision in the same way. And of course many other possibilities.

$\endgroup$
2
  • $\begingroup$ Thanks for the great answer. I still have two questions: 1) what if we can observe not the true state but a stochastic manifestation of it - e.g. a coin toss has an unobservable probability $\theta$ of turning head but we only observe the past history of the coin toss. It can still be formulated as a supervised learning problem (say, there are many coins and we can describe them with features $X$), and the loss function will still be L(head/tail, prediction)? Another example is to predict which team will win a sports match. $\endgroup$ Sep 5, 2021 at 16:14
  • $\begingroup$ 2) When the true state can be observed, i.e. $\theta$ is the same as $y$, I wonder how some of the concepts of "risk" in statistical decision theory carry over. Most importantly, the concept of posterior expected loss defined as $E[L(\theta, \delta(X))|X]$ with the expectation taken over the posterior $P(\theta|X)$ seems meaningless, as now $\theta$ is an observed truth. The Bayes risk, $E[L(\theta, \delta(X))]$, with the expectation taken over the joint distribution of $(\theta, X)$ is another something I no longer feel I understand in the ML setting. $\endgroup$ Sep 5, 2021 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.