"Dumb" log-loss for a binary classifier

Question

I am trying to understand how I can best compare a classifier that I have trained and tuned against a "dumb" classifier, particularly in the context of binary classification with imbalanced classes.

Here's a summary of my experiment: suppose I have a dataset that I have split in training/validation/test sets. My classifier (random forest or gradient boosted trees here) is trained on the training set and then the best hyperparameters are selected by evaluating the log-loss on the validation set and choosing the model with the lowest log-loss. I am choosing here to minimise the log-loss because I want the model to incur a very high penalty when it confidently predicts the wrong class.

Suppose now that I want to compare my classifier against a dumb classifier on the test set and suppose that I have the following class proportions in the test set:
55% class 0 and 45% class 1.
Suppose also that the accuracy of my classifier is 60%.

A particularly dumb classifier is one that would for example classify everything as 0, which would result in an accuracy of 55%. This would be of course a poor benchmark to use, even as a dumb baseline.

Say I want to compare my classifier against a random one, for example a biased coin. How do I calculate the log-loss of the biased coin?

I thought initially that I should use the proportion of classes in the test set as the probabilities of the biased coin i.e. $$P[X=0]=0.55 \text{ and } P[X=1]=0.45$$ This would result in a log-loss of :
$$-\frac{1}{N}\sum_{i=1}^{N}(y_i\log(p_i) + (1-y_i)\log(1-p_i))=-(0.55\log(0.55) + 0.45\log(0.45))=0.6881$$

In doing so, I feel like I am making a mistake since it would be like this random classifier already knows the distribution of my test set? Is the correct thing to assume that the probabilities of my biased coin classifier are the ones observed in the training set (for example 51% class 0 and 49% class 1) and simulate randomly the choice of either class 0 or 1 on my test set using those probabilities so as to calculate a log-loss?

Thanks!

Answer 1

You are correct, if your "dumb" classifier knows the frequency of successes in the test set, it in fact works as an oracle, and is not that dumb. You're leaking the data from test set. It is easy to imagine an extreme case with big discrepancy between train and test set where such a "dumb" classifier would in fact outperform the model trained using only the train set.

What you should do is to base your "dumb" classifier on the distribution of the train dataset. In fact, for binary data, predicting the mean, or probability of success, is the best single-value prediction you can make assuming squared error or log-loss, so it is a pretty nice benchmark of the simplest but not completely useless model.

Answer 2

I like my explanation of $R^2$ here and how it relates to a naïve model. You would be looking for the McFadden’s $R^2$ that I mention, as that compares the log loss of your model to that of one that naively predicts the prior probability every time, much as linear regression $R^2$ naively guesses the pooled/marginal mean $\bar y$ every time in the “denominator”.

$$R^2_{McFadden}= 1- \dfrac{ \sum \big( y_i\log(\hat p_i)+ (1-y_i)\log(1-\hat p_i) \big) }{ \sum\big( y_i\log(p_{train})+ (1-y_i)\log(1-p_{train}) \big) } $$

The denominator is (proportional to) the log loss of the biased coin that you described.

Positive values of this quality represent an improvement over the “dumb” classifier. We take the $p_{train}$ in the denominator to use our knowledge of the training set without cheating and looking at any of the out-of-sample data.

EDIT

If you see me write about $R^2$ on here, you'll see me use that word "naïve". I see it this way:

You have to predict the probability that a photo is of a dog or a cat. The obvious move would be to look at the photo and decide how likely it is to be of a dog or of a cat. I, however, will not show you the picture, but I will tell you that there are as many dog photos as cat photos (or ten times as many dogs as cats, or whatever the ratio is).

Knowing nothing about the photo, the sensible, even if naïve, guess is that class ratio (the "prior" probability of being a dog or being a cat). If there are as many dog pictures as cat pictures, guess that there's a $50/50$ chance of either. If there are nine dog photos for every one cat photo, guess that there is a $90\%$ chance of being a dog and a $10\%$ chance of being a cat.