Terminology question: distinguishing two meanings of "loss function"?

Question

I've heard people use "loss function" to refer to 2 different functions, with different type signatures:

1) A real-valued function of a label, $y$, and a prediction $\hat{y}$.

2) A real-valued function of a parameter $\theta$; this function depends on the loss function (in the first sense) and the data distribution.

I'd like to talk about both of these things, and have good terminology for distinguishing them. Any suggestions?

My current inclination would be to call (2) a "task", but I think that's not ideal.

EDIT: Here's an example, to clarify what I mean by (2). Suppose we have data distribution P(X,Y), and are using MSE. Then the loss is: $$L(\theta) = E_{(x,y) \sim P(X,Y)} (y - f_\theta(x))^2$$ In this example, I mean $L(\theta)$.

Answer 1

In statistical parlance, there is usually:

1. A loss function $\ell(\hat y, y)$, e.g. squared loss $\frac12 (\hat y - y)^2$, 0-1 loss $\mathbf{1}(\hat y \ne y)$, or so on, which determines the loss due to a prediction $\hat y$ for the truth $y$.
2. A risk, or expected loss, function $L(\theta) = \mathbb E_{(x,y)}[ \ell( f_\theta(x), y ) ]$, which tells you the expected loss of a predictor $f_\theta$.

In machine learning, the latter function $L(\theta)$ is also sometimes called a loss (e.g. in talking about the loss surface of a model). If you need to disambiguate, in predictive contexts I think risk might not generally be immediately understood; expected loss might, but it might also conjure associations of being e.g. the expected value of $L$ after running a randomized training procedure.

Objective function might work for $L(\theta)$, but that would typically also include any explicit regularization terms you're using, which might not be what you want.

In short: I don't thing there's a totally unambiguous term that will be immediately understood. You can call it any of these things, particularly if you're not also referring to the function $\ell$; just make sure to clarify what you're talking about on the first usage.

Answer 2

The first is the loss function of the predictions, and the second is the loss function of the parameters.

In the energy-based model literature, the second one is often called the loss functional because, as you point out, it's actually a function of a function.

You could also define a loss function of the inputs or intermediate features and the parameters, or without the parameters if you hold them fixed.