Given a classification problem with k classes, suppose a model outputting a probability distribution over the classes that uses some gradient-based learning method is used (yes, in my case it's neural networks, but I guess it could be any). A typical choice of loss for classification is cross-entropy, but let's say we use L2 loss (with the predicted probability distribution and the one-hot encoded label). What is the relationship between the L2 loss value and the accuracy of the prediction? (I consider the accuracy as the probability value for the right class in the prediction).
Obviously, given a L2 loss value there are several accuracy values possible, but, after doing some plots with synthetic random examples, it seems that there is a very strong correlation.
It seems there is some low-variance probability distribution there, but I don't know which.
PD: I'm not sure whether this fits better here or in math.