Loss function in Supervised Learning vs Statistical Decision Theory I am confused by the different definitions of Loss Function in statistical decision theory vs machine learning.
In statistical decision theory, a loss function is typically defined as $L(\theta, \delta(X))$, where $\theta$ is the true, unknown parameter, $\delta(.)$ is the decision rule, and $X$ is data (generated from $\theta$?). See for example lectures of the Theory of Statistics class.
In machine learning, it seems the loss function is defined as $L(y, f(X))$, where $y$ is the true label and $f(x)$ is some model. See, for example, Elements of Statistical Learning chapter 2.4.
My question is if they are talking about the same thing. It seems different. For example, if I am going to predict the next coin toss of an unknown coin, I can then model the coin toss as following a Bernoulli distribution with an unknown parameter $\theta$.
Let $X$ be some historical data. Then it appears that the loss function from statistical decision theory is computing my prediction $\delta(X)$ against the unknown parameter $\theta$ whereas in ML, it is computing the same prediction $\delta(X)$ (or $f(X)$) against the true label?
I am having trouble reconciling the two concepts.
 A: I would say this is more a difference in the form of the decision than the loss. The loss function in both cases is Loss(true state of nature, your decision), but it simplifies differently depending on the form of the decision
In point prediction settings (such as a lot of ML), the decision is a potential value of the label, and the state of nature effectively simplifies to the true value of the label, so the loss $L(y, \hat y)$ can be written as the loss from predicting $\hat y$ when the truth is $y$.
In parametric inference settings, the decision is a potential value of the parameter, and the state of nature effectively simplifies to the true parameter value, so the loss $L(\theta, \hat\theta)$ can be written as the loss from estimating $\hat\theta$ when the truth is $\theta$.
There are more complicated settings, too. For example, your decision might be an interval, and the state of nature might be a value, and the loss could be the length of the interval plus the distance from the value to the closest point of the interval (possibly zero)[PDF]. In that setting there isn't the nice correspondence between potential decisions and potential states of nature, and the loss doesn't simplify down to a summary of the error in the decision in the same way. And of course many other possibilities.
