Understanding StatQuest video: why cross entropy is used over Sum Squared Error

Question

I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses:

He the says that loss is very high for CE than MSE. But I am not able to get how he got that orange straight line. I tried to plot in desmos.

. Note that what I have plotted in blue is just (1-p), which I dont believe Josh thought of for MSE. In fact, I dont know if my Cross Entropy plot is correct. Josh says its -log(p). But the x axis is not clearly labeled as p by Josh. What he exactly means by X axis labels "worst prediction" and "great prediction"? How exactly make sense out of this graph by Josh?

PS: I didnt plot $(1-p)^2$ because its not straight line as in video.

Answer 1

$p$ here is the predicted probability of an outcome to belong to the class it actually turns out to belong to. To illustrate: suppose we have three possible classes, and our model says that a given new instance has a probability $0.5$ of belonging to the first class, $0.3$ of belonging to the second, and $0.2$ of belonging to the third class. The instance actually turns out to belong to the second class. So in this case, $p=0.3$.

Now, the question is one of how to measure the "badness" of our probabilistic prediction. The two losses you note are quite common. We note that a perfect prediction would have been $p=1$, i.e., we would have predicted the new instance to belong to the class it actually turned out to belong to with certainty. (Of course, this also implies that our probabilistic prediction of the instance to belong to the other classes would have been zero.) Both losses are zero for such perfect predictions:

$$ -\log p = -\log 1 = 0\quad\text{and}\quad (1-p)^2=(1-1)^2=0. $$

The question is how these losses behave when our prediction is not perfect. Of course, the lower $p$, the worse the prediction is - and correspondingly, the higher the loss should be.

This is what is depicted. The MSE grows quadratically with decreasing $p$, and ends up at $1$ if $p=0$ (i.e., if we had believed it to be impossible for the instance to belong to the class it actually does belong to). So the MSE is bounded. The line should indeed not be straight, but a parabola. This is presumably to draw attention to the fact that the Cross-Entropy grows much faster as $p$ decreases. In fact, the Cross-Entropy is infinite for $p=0$: seeing an outcome we believed to be impossible yields an infinitely large Cross-Entropy loss.

Cross-Entropy is also known as the log loss, and the MSE is the Brier score. (Many concepts are developed independently by different people, and given different names.) Both are proper scoring rules, i.e., losses that incentivize us to output the true class membership probabilities. (Other losses, like accuracy, do not. Also, a straight line loss like you plotted, which would correspond to the Mean Absolute Error, or MAE, would reward biased predictions.) There is a comparison between the two losses at Why is LogLoss preferred over other proper scoring rules?