4
$\begingroup$

Given a classification problem with k classes, suppose a model outputting a probability distribution over the classes that uses some gradient-based learning method is used (yes, in my case it's neural networks, but I guess it could be any). A typical choice of loss for classification is cross-entropy, but let's say we use L2 loss (with the predicted probability distribution and the one-hot encoded label). What is the relationship between the L2 loss value and the accuracy of the prediction? (I consider the accuracy as the probability value for the right class in the prediction).

Obviously, given a L2 loss value there are several accuracy values possible, but, after doing some plots with synthetic random examples, it seems that there is a very strong correlation.

L2 loss vs accuracy

It seems there is some low-variance probability distribution there, but I don't know which.

PD: I'm not sure whether this fits better here or in math.

$\endgroup$
1

1 Answer 1

1
$\begingroup$

There should be a relationship between $\ell_2$ loss and accuracy. The $\ell_2$ loss measures the difference between the predicted probabilities of class membership and the actual class membership. As these predicted probabilities get closer to the actual class, the $\ell_2$ loss is going to decrease, and the probability of the true class having the highest probability will increase.

In the extreme, when all predicted probabilities are either zero or one and assign the probability of one to the correct category, accuracy will be perfect, and $\ell_2$ loss will be perfect. However, you could tweak the probability to give only $0.7$ probability of the correct class and $0.3$ probability distributed between the other classes, and this would raise the $\ell_2$ loss without affecting the accuracy (since the correct category still has the highest probability), so $\ell_2$ and accuracy do not have to move in perfect unison.

I like the analogy here about sprinters having strong legs, yet leg strength not being a perfect proxy of sprinting speed.

$\endgroup$
1
  • $\begingroup$ Thanks, and good reference in the link. I understand your explanation, although I was wondering if there was more of a formal relationship between the two, beyond the reasoning about the correlation. But, like you said, the two do not need to go exactly hand in hand, so I suppose there is no general rule that relate them. $\endgroup$
    – jdehesa
    Apr 9, 2023 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.