I'm trying to better understand log loss and how it works but one thing I can't seem to find is putting the log loss number into some sort of context. If my model has a log loss of 0.5, is that good? What's considered a good and bad score? How do these thresholds change?
The logloss is simply $L(p_i)=-\log(p_i)$ where $p$ is simply the probability attributed to the real class.
So $L(p)=0$ is good, we attributed the probability $1$ to the right class, while $L(p)=+\infty$ is bad, because we attributed the probability $0$ to the actual class.
So, answering your question, $L(p)=0.5$ means, on average, you attributed to the right class the probability $p\approx0.61$ across samples.
Now, deciding if this is good enough is actually application-dependent, and so it's up to the argument.
Like any metric, a good metric is the one better that the "dumb", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.
This "dumb"-guess depends on 2 factors :
- the number of classes
- the balance of classes : their prevalence in the observed dataset
In the case of the LogLoss metric, one usual "well-known" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting
p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict
p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting
0.5 will be dumber.
I. Impact of the number of classes
N on the dumb-logloss:
In the balanced case (every class has the same prevalence), when you predict
p = prevalence = 1 / N for every observation, the equation becomes simply :
Logloss = -log(1 / N)
Ln, neperian logarithm for those who use that convention.
In the binary case,
N = 2 :
Logloss = - log(1/2) = 0.693
So the dumb-Loglosses are the following :
II. Impact of the prevalence of classes on the dumb-Logloss:
a. Binary classification case
In this case, we predict always
p(i) = prevalence(i), and we obtain the following table :
So, when classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use
b. Three-class case
"Dumb"-logloss depending on prevalence - three-class case :
We can see here the values of balanced binary and three-class cases (0.69 and 1.1).
A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.
Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.
In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.
So this is actually more complicated than Firebugs response and it all depends on the inherent variation of the process you are trying to predict.
When I say variation what I mean is 'if an event was to repeat under the exact same conditions, known and unknown, what's the probability that the same outcome will occur again'.
A perfect predictor would have a loss, for probability P: Loss = P ln P + (1-P) ln (1-P)
If you are trying to predict something where, at its worse, some events will be predicted with an outcome of 50/50, then by integrating and taking the average the average loss would be: L=0.5
If what you are trying to predict is a tad more repeatable the loss of a perfect model is lower. So for example, say with sufficient information a perfect model was able to predict an outcome of an event where across all possible events the worst it could say is 'this event will happen with 90% probability' then the average loss would be L=0.18.
There is also a difference if the distribution of probabilities is not uniform.
So in answer to your question the answer is 'it depends on the nature of what you are trying to predict'
I'd say the standard statistics answer is to compare to the intercept only model. (this handles the unbalanced classes mentioned in other answers) cf mcFadden's pseudo r^2. https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/
Now the problem is what the maximum value is. fundamentally the problem is that probability of an event is undefined outside a model for the events. the way I would suggest is that you take your test data and aggregate it to a certain level, to get a probability estimate. then calculate the logloss of this estimate.
eg you are predicting click through rate based on (web_site, ad_id, consumer_id), then you aggregate clicks, impressions to eg web_site level and calculate the ctr on the test set for each web site. then calculate log_loss on your test data_set using these test click through rates as predictions. This is then the optimal logloss on your test set for a model only using website ids. The problem is we can make this loss as small as we like by just adding more features until each record is uniquely identified.