0
$\begingroup$

I am training a neural network for a multi-class classification task. The loss function I am using is the CrossEntropyLoss implemented in pytorch, which is, according to the documents, a combination of logsoftmax and negative log likelihood loss (forgive me for not knowing much about them, all I know is that cross entropy is frequently used for classification). I started out with a loss of 2.4 and then it descends to and fluctuates in the range 0.65~0.7. Can I infer from this value how accurate my model is now? I remember reading in an article that a cross entropy loss of 0.69 is equal to random guess in the binary classification case, is it different somehow in the multi-class classification? Or does it mean my model is basically guessing randomly?

More details, if they matter:

  • I am running on CIFAR10, so there are 10 classes.

  • The loss is calculated per sample, so the batch size is not reflected here.

$\endgroup$
1
$\begingroup$

is it different somehow in the multi-class classification? Or does it mean my model is basically guessing randomly?

You have to understand how you get the score of a random baseline.

Suppose you have a binary classification problem where classes are balanced, that is, the pmf of the labels in your dataset is $q_i=1/2$. If you do random guesses, you don't even look at the inputs (images in your case) and assign probability $p_i=1/2$ to the two classes all the time. The cross-entropy you obtain is $H[q,p] = - \sum_i q_i \log p_i = - 2 * 1/2 * \log(1/2) = \log(2) = 0.69$ (using the natural log).

From there, you can generalise this reasoning in the 10-classes case by yourself. In the unbalanced case (CIFAR10 is balanced so you won't need that, but I'm still explaining), of course you simply change the $q_i$.

Understanding the above, you 1) understand why the model starts out with a loss around $2.4$ and 2) know whether you are doing better than guessing at random. Leave a comment if that is not the case.

Can I infer from this value how accurate my model is now?

Accuracy has a precise meaning in evaluation of classification model and I don't know if that's what you mean. You can not infer accuracy from the cross-entropy. Accuracy is defined as the proportion of labels that are correctly classified by your classifier on the test set. How do you "classify" using a probabilistic model? You choose the class that has the highest probability, this is called the Bayes classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.