How low does the cross entropy loss need to be for me to be confident in my model?

Question

I am training a neural network for a multi-class classification task. The loss function I am using is the CrossEntropyLoss implemented in pytorch, which is, according to the documents, a combination of logsoftmax and negative log likelihood loss (forgive me for not knowing much about them, all I know is that cross entropy is frequently used for classification). I started out with a loss of 2.4 and then it descends to and fluctuates in the range 0.65~0.7. Can I infer from this value how accurate my model is now? I remember reading in an article that a cross entropy loss of 0.69 is equal to random guess in the binary classification case, is it different somehow in the multi-class classification? Or does it mean my model is basically guessing randomly?

More details, if they matter:

• I am running on CIFAR10, so there are 10 classes.

• The loss is calculated per sample, so the batch size is not reflected here.

Answer 1

is it different somehow in the multi-class classification? Or does it mean my model is basically guessing randomly?

You have to understand how you get the score of a random baseline.

Suppose you have a binary classification problem where classes are balanced, that is, the pmf of the labels in your dataset is $q_i=1/2$. If you do random guesses, you don't even look at the inputs (images in your case) and assign probability $p_i=1/2$ to the two classes all the time. The cross-entropy you obtain is $H[q,p] = - \sum_i q_i \log p_i = - 2 * 1/2 * \log(1/2) = \log(2) = 0.69$ (using the natural log).

From there, you can generalise this reasoning in the 10-classes case by yourself. In the unbalanced case (CIFAR10 is balanced so you won't need that, but I'm still explaining), of course you simply change the $q_i$.

Understanding the above, you 1) understand why the model starts out with a loss around $2.4$ and 2) know whether you are doing better than guessing at random. Leave a comment if that is not the case.

Can I infer from this value how accurate my model is now?

Accuracy has a precise meaning in evaluation of classification model and I don't know if that's what you mean. You can not infer accuracy from the cross-entropy. Accuracy is defined as the proportion of labels that are correctly classified by your classifier on the test set. How do you "classify" using a probabilistic model? You choose the class that has the highest probability, this is called the Bayes classifier.

Answer 2

The following chart demonstrates distribution of cross-entropy for random guessing based on the label distribution. It is clear that in the middle of the chart the cross-entropy reaches its maximum (.69) as there the randomness is maximum too.

[0.1985152433458726,
0.3250829733914482,
 0.4227090878059909,
 0.5004024235381879,
 0.5623351446188083,
 0.6108643020548935,
 0.6474466390346325,
 0.6730116670092565,
 0.6881388137135884,
 0.6931471805599453,
 0.6881388137135884,
 0.6730116670092564,
 0.6474466390346324,
 0.6108643020548934,
 0.5623351446188082,
 0.5004024235381879,
 0.4227090878059907,
 0.325082973391448,
 0.19851524334587234]

Code:

import numpy as np
def cross_entropy_by_dist(positive_percent):
    negative_percent = 1-positive_percent
    sc = positive_percent*np.log(positive_percent)+negative_percent*np.log(negative_percent)
    return -sc

out = []
for i in np.arange(0.05, 1, .05):
    out.append(cross_entropy_by_dist(i))
    
import matplotlib.pyplot as plt
plt.plot(np.arange(.05, 1, .05), out)
plt.grid()