Relation between L2 loss and accuracy

Question

Given a classification problem with k classes, suppose a model outputting a probability distribution over the classes that uses some gradient-based learning method is used (yes, in my case it's neural networks, but I guess it could be any). A typical choice of loss for classification is cross-entropy, but let's say we use L2 loss (with the predicted probability distribution and the one-hot encoded label). What is the relationship between the L2 loss value and the accuracy of the prediction? (I consider the accuracy as the probability value for the right class in the prediction).

Obviously, given a L2 loss value there are several accuracy values possible, but, after doing some plots with synthetic random examples, it seems that there is a very strong correlation.

It seems there is some low-variance probability distribution there, but I don't know which.

PD: I'm not sure whether this fits better here or in math.

Answer 1

There should be a relationship between $\ell_2$ loss and accuracy. The $\ell_2$ loss measures the difference between the predicted probabilities of class membership and the actual class membership. As these predicted probabilities get closer to the actual class, the $\ell_2$ loss is going to decrease, and the probability of the true class having the highest probability will increase.

In the extreme, when all predicted probabilities are either zero or one and assign the probability of one to the correct category, accuracy will be perfect, and $\ell_2$ loss will be perfect. However, you could tweak the probability to give only $0.7$ probability of the correct class and $0.3$ probability distributed between the other classes, and this would raise the $\ell_2$ loss without affecting the accuracy (since the correct category still has the highest probability), so $\ell_2$ and accuracy do not have to move in perfect unison.

I like the analogy here about sprinters having strong legs, yet leg strength not being a perfect proxy of sprinting speed.