4
$\begingroup$

I've heard people use "loss function" to refer to 2 different functions, with different type signatures:

1) A real-valued function of a label, $y$, and a prediction $\hat{y}$.

2) A real-valued function of a parameter $\theta$; this function depends on the loss function (in the first sense) and the data distribution.

I'd like to talk about both of these things, and have good terminology for distinguishing them. Any suggestions?

My current inclination would be to call (2) a "task", but I think that's not ideal.

EDIT: Here's an example, to clarify what I mean by (2). Suppose we have data distribution P(X,Y), and are using MSE. Then the loss is: $$L(\theta) = E_{(x,y) \sim P(X,Y)} (y - f_\theta(x))^2$$ In this example, I mean $L(\theta)$.

$\endgroup$
3
  • $\begingroup$ It's unclear what (2) refers to: could you clarify? If you're not sure--and even then--please review some of our answers about loss functions. $\endgroup$
    – whuber
    Dec 6, 2019 at 20:02
  • $\begingroup$ If for example the function of $\theta$ being high is worse than being low then this might be a penalty and your aim might be to minimise the sum of the loss and the penalty, perhaps calling this sum the cost $\endgroup$
    – Henry
    Dec 6, 2019 at 20:13
  • $\begingroup$ @whuber I clarified. $\endgroup$ Dec 22, 2019 at 22:31

2 Answers 2

1
$\begingroup$

In statistical parlance, there is usually:

  1. A loss function $\ell(\hat y, y)$, e.g. squared loss $\frac12 (\hat y - y)^2$, 0-1 loss $\mathbf{1}(\hat y \ne y)$, or so on, which determines the loss due to a prediction $\hat y$ for the truth $y$.
  2. A risk, or expected loss, function $L(\theta) = \mathbb E_{(x,y)}[ \ell( f_\theta(x), y ) ]$, which tells you the expected loss of a predictor $f_\theta$.

In machine learning, the latter function $L(\theta)$ is also sometimes called a loss (e.g. in talking about the loss surface of a model). If you need to disambiguate, in predictive contexts I think risk might not generally be immediately understood; expected loss might, but it might also conjure associations of being e.g. the expected value of $L$ after running a randomized training procedure.

Objective function might work for $L(\theta)$, but that would typically also include any explicit regularization terms you're using, which might not be what you want.

In short: I don't thing there's a totally unambiguous term that will be immediately understood. You can call it any of these things, particularly if you're not also referring to the function $\ell$; just make sure to clarify what you're talking about on the first usage.

$\endgroup$
1
$\begingroup$

The first is the loss function of the predictions, and the second is the loss function of the parameters.

In the energy-based model literature, the second one is often called the loss functional because, as you point out, it's actually a function of a function.

You could also define a loss function of the inputs or intermediate features and the parameters, or without the parameters if you hold them fixed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.