I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the partial derivative of the cross-entropy loss with respect to the predicted probabilities. Here's where my confusion lies:

Why is it that $ \frac{\partial}{\partial f(\mathbf{x})_c}(-\log f(\mathbf{x})_y) = \frac{-1_{(y=c)}}{f(\mathbf{x})_y} \quad$? ($1_{(y=c)}=1$if $y=c$, otherwise 0)

Given that $ f(\mathbf{x})_c = p(y=c|\mathbf{x}) $ and knowing that the sum of probabilities across all classes equals one, $ \sum_c f(\mathbf{x})_c = p(y=c|\mathbf{x}) = 1 $, it seems there should be a relationship between the derivatives across different classes. Thus, shouldn't the derivative $ \frac{\partial}{\partial f(\mathbf{x})_c}(-\log f(\mathbf{x})_y) $ be equivalent to $ \frac{\partial}{\partial f(\mathbf{x})_c}( (1-\sum_{c' \neq y}f(\mathbf{x})_{c'}))=-\frac{1}{ f(\mathbf{x})_y}\quad\frac{\partial}{\partial f(\mathbf{x})_c}( (1-\sum_{c' \neq y}f(\mathbf{x})_{c'})) $? And wouldn't this not equal zero (for $c\neq y$, this equals to -1), thereby presenting a contradiction?

I'm trying to wrap my head around this concept and would greatly appreciate any insights or explanations you might offer.


1 Answer 1


The short answer is your intuition is correct only when the true probability of every class is never 0. In single-label classification, we are dealing with almost the opposite case, every class except one "correct" one has a true probability of 0, while the correct one has a probability of 1.

The longer answer is:

You are using formula from a context where the true probability distribution is a value of 1 at a single point and 0 everywhere else. E.g., in a typical single-label classification task, but trying to interpret it with intuition from a context where the true probability distribution is always non-zero.

Imagine we have to classify a bunch of images of dogs, cats, and birds.

The cross entropy for a discrete probability distribution is given by:

$$ H(p, q) = -\sum_{c \in \mathcal{C}}p(c)\text{ log }q(c) $$

Where $p(c)$ is the true probability distribution and $q(c)$ is the predictions coming from your model. $C$ is the set of all possible classes (dog, cat, and bird).

We are assuming that for each picture, there really only is 1 correct answer. E.g., it's impossible for the true classification of a picture to be 20% bird and 80% dog.

In that case, $p(x)$ must have the form $1_{(c=y)}$. For a picture of a dog, $p(dog) = 1$ and $p(cat) = p(bird) = 0$.

That means we can rewrite the cross entropy loss as:

$$ -\sum_{c \in \mathcal{C}}p(c)\text{ log }q(c) = -1_{(c = y)} \text{log } q(c) = -\text{log } q(c_y) $$

This means that if your model thinks that a picture of a dog is 50% dog, 49% cat, and 1% bird, it will have the exact same cross entropy loss as a model that thinks that picture of the dog is 50% dog, 25% cat, and 25% bird. This should give some intuition on why the derivatives with respect to the other classes don't matter. When you multiply your model probabilities of the wrong classes with the true probabilities of the wrong classes, 0, you'll always just get 0, and no meaningful information about your model's agreement with the true class probabilities will be present.

One more example is to imagine that we are classifying cards in a deck. We set our model's space of possible classes to include all the regular cards but also a special "11 of hearts" card that doesn't exist (true probability is 0). Imagine then that we are trying to classify a 5 of clubs. We try to make use of your approach looking at just $1 - \sum_{c' \neq y}$. This will ultimately not work because over that summation, there will be that one moment where we multiply our model's probability that the card is a 11 of hearts (which could be anything from 0 to 1) with our true probability that the card is an 11 of hearts (which is 0). The moment that happens, it will be impossible for us to know how our model's combined probability mass at the 5 of clubs (not included in the summation) vs. 11 of hearts (information lost by multiplication with 0) was distributed between those two cards.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.