Is this notation , $\ell(y,f(x;\theta))= -\log p(y|f(x;\theta))$, correct for the negative log probability loss function of a classifier?

Question

In Murphy’s Probabilistic Machine Learning: An Introduction, he states that the loss function for a probabilistic classifier $f(x;\theta)$ is the following: $$\ell(y,f(x;\theta))= -\log p(y \mid f(x;\theta))$$ where $x$ is the input, $y$ the corresponding true output and $\theta$ the parameter vector of the model $f$.

My question is one of notation: Should it not rather be $-\log p(y \mid f,x,\theta )$? I feel that the original notation is saying the probability of the true label given the model output $f(x;\theta)$ taking a specific value $f(x;\theta)$ which doesn’t really capture what the “meaning” of this the probability distribution in this loss function (the probability of the true label $y$ of $x$, given the model choice $f$, the input $x$, and the parameters of the model).

Am I splitting hairs / missing something?

Answer 1

It doesn’t really matter which notation, though there’s a subtle reason you might prefer Murphy’s.

We could create an arbitrary variable $u$ of the correct type and measure the loss $\ell(y, u)$. Now Murphy’s version of the loss is $- \log p(y \mid u)$. Your proposed formula requires you to ‘remember’ how $u$ was created. That doesn’t make it wrong, but Murphy’s removes that assumption.

---

I did want to clear up one thing, though.

I feel that the original notation is saying the probability of the true label given the model output 𝑓(𝑥;𝜃) taking a specific value 𝑓(𝑥;𝜃) which doesn’t really capture what the “meaning” of this the probability distribution in this loss function

I don’t agree with the part that I italicized.

As a reminder, the probabilistic classifiers that Murphy describes here produce a probability distribution over classes, not a single ‘best’ class. For example, in the three-class iris dataset he describes, $f(x; \theta)$ is not $\text{setosa}$; it’s $\{\text{virginica} \mapsto 0.3, \text{setosa} \mapsto 0.6, \text{versicolor} \mapsto 0.1\}$, if you’ll forgive the notation.

That means that $f(x; \theta)$ has all of the information that you need to compute the probability of $y$ given $f$, $\theta$, and $x$. You no longer need access to the three ‘ingredients’. Instead, $\ell$ compares the ‘true’ label $y$ with the induced distribution $u=f(x; \theta)$.

---

For people who find this question in the future and are still curious, this definition of the loss function is Equation (1.8) of Murphy’s Probabilistic Machine Learning: An Introduction. It is on page 7 of the draft from March 8, 2021.