2
$\begingroup$

In Murphy’s Probabilistic Machine Learning: An Introduction, he states that the loss function for a probabilistic classifier $f(x;\theta)$ is the following: $$\ell(y,f(x;\theta))= -\log p(y \mid f(x;\theta))$$ where $x$ is the input, $y$ the corresponding true output and $\theta$ the parameter vector of the model $f$.

My question is one of notation: Should it not rather be $-\log p(y \mid f,x,\theta )$? I feel that the original notation is saying the probability of the true label given the model output $f(x;\theta)$ taking a specific value $f(x;\theta)$ which doesn’t really capture what the “meaning” of this the probability distribution in this loss function (the probability of the true label $y$ of $x$, given the model choice $f$, the input $x$, and the parameters of the model).

Am I splitting hairs / missing something?

$\endgroup$
2
$\begingroup$

It doesn’t really matter which notation, though there’s a subtle reason you might prefer Murphy’s.

We could create an arbitrary variable $u$ of the correct type and measure the loss $\ell(y, u)$. Now Murphy’s version of the loss is $- \log p(y \mid u)$. Your proposed formula requires you to ‘remember’ how $u$ was created. That doesn’t make it wrong, but Murphy’s removes that assumption.


I did want to clear up one thing, though.

I feel that the original notation is saying the probability of the true label given the model output 𝑓(𝑥;𝜃) taking a specific value 𝑓(𝑥;𝜃) which doesn’t really capture what the “meaning” of this the probability distribution in this loss function

I don’t agree with the part that I italicized.

As a reminder, the probabilistic classifiers that Murphy describes here produce a probability distribution over classes, not a single ‘best’ class. For example, in the three-class iris dataset he describes, $f(x; \theta)$ is not $\text{setosa}$; it’s $\{\text{virginica} \mapsto 0.3, \text{setosa} \mapsto 0.6, \text{versicolor} \mapsto 0.1\}$, if you’ll forgive the notation.

That means that $f(x; \theta)$ has all of the information that you need to compute the probability of $y$ given $f$, $\theta$, and $x$. You no longer need access to the three ‘ingredients’. Instead, $\ell$ compares the ‘true’ label $y$ with the induced distribution $u=f(x; \theta)$.


For people who find this question in the future and are still curious, this definition of the loss function is Equation (1.8) of Murphy’s Probabilistic Machine Learning: An Introduction. It is on page 7 of the draft from March 8, 2021.

$\endgroup$
2
  • $\begingroup$ Why does the my proposed formula require us to remember how $u$ was “created”? $\endgroup$ – apprentice9 Apr 6 at 18:31
  • $\begingroup$ It explicitly includes the ingredients $f$, $\theta$, and $x$ in the formula. $p$ is a function of four variables, not two. $\endgroup$ – Arya McCarthy Apr 6 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.