I'm trying to better understand log loss and how it works but one thing I can't seem to find is putting the log loss number into some sort of context. If my model has a log loss of 0.5, is that good? What's considered a good and bad score? How do these thresholds change?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ This thread is relevant here: stats.stackexchange.com/q/414349/121522 $\endgroup$– mktCommented Jun 24, 2019 at 11:27
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
The logloss is simply $L(p_i)=-\log(p_i)$ where $p$ is simply the probability attributed to the real class.
So $L(p)=0$ is good, we attributed the probability $1$ to the right class, while $L(p)=+\infty$ is bad, because we attributed the probability $0$ to the actual class.
So, answering your question, $L(p)=0.5$ means, on average, you attributed to the right class the probability $p\approx0.61$ across samples.
Now, deciding if this is good enough is actually application-dependent, and so it's up to the argument.
$\begingroup$
$\endgroup$
3
Like any metric, a good metric is the one better that the "dumb", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.
This "dumb"-guess depends on 2 factors :
- the number of classes
- the balance of classes : their prevalence in the observed dataset
In the case of the LogLoss metric, one usual "well-known" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.
I. Impact of the number of classes N on the dumb-logloss:
In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :
Logloss = -log(1 / N)
log being Ln, neperian logarithm for those who use that convention.
In the binary case, N = 2 : Logloss = - log(1/2) = 0.693
So the dumb-Loglosses are the following :
II. Impact of the prevalence of classes on the dumb-Logloss:
a. Binary classification case
In this case, we predict always p(i) = prevalence(i), and we obtain the following table :
So, when classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use
b. Three-class case
"Dumb"-logloss depending on prevalence - three-class case :
We can see here the values of balanced binary and three-class cases (0.69 and 1.1).
CONCLUSION
A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.
Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.
In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.
-
$\begingroup$ (+1) Welcome to CV! You can use math typesetting in your post. More information: math.meta.stackexchange.com/questions/5020/… $\endgroup$– Sycorax ♦Commented Mar 5, 2019 at 15:23
-
2
-
$\begingroup$ Great Answer Federico! Maybe I would just mention that in the binary unbalanced case p(i) = percentage of the positive class $\endgroup$ Commented Oct 17, 2022 at 12:39
$\begingroup$
$\endgroup$
3
So this is actually more complicated than Firebugs response and it all depends on the inherent variation of the process you are trying to predict.
When I say variation what I mean is 'if an event was to repeat under the exact same conditions, known and unknown, what's the probability that the same outcome will occur again'.
A perfect predictor would have a loss, for probability P: Loss = P ln P + (1-P) ln (1-P)
If you are trying to predict something where, at its worse, some events will be predicted with an outcome of 50/50, then by integrating and taking the average the average loss would be: L=0.5
If what you are trying to predict is a tad more repeatable the loss of a perfect model is lower. So for example, say with sufficient information a perfect model was able to predict an outcome of an event where across all possible events the worst it could say is 'this event will happen with 90% probability' then the average loss would be L=0.18.
There is also a difference if the distribution of probabilities is not uniform.
So in answer to your question the answer is 'it depends on the nature of what you are trying to predict'
-
$\begingroup$ It's well known a $L\approx 0.693$ is the non-informative binary logarithmic loss (i.e. random guessing). I don't really follow your calculations, are you assuming an uniform distribution for $p$ or something like that? $\endgroup$– FirebugCommented Aug 14, 2017 at 11:48
-
$\begingroup$ Imagine you have a system, where you know all the possible information about its current state. Imagine that system has some internal level of randomness so that, given all the parameters about its state, an outcome can be different. Say for example in this system it can range from 0-10%. A perfect model (ie one where its performance is only limited by the inherent variation) would get $L\approx 0.18$. For comparison a system which varies from 0-100% would get at its best $L\approx 0.5$. $\endgroup$– simeonCommented Aug 14, 2017 at 12:05
-
1$\begingroup$ I don't disagree with your worse case. I am merely saying that a 'good result' depends on the system. 0.4 may indicate a good result for some systems (like the latter example) or a bad one (for the former). $\endgroup$– simeonCommented Aug 14, 2017 at 12:07
$\begingroup$
$\endgroup$
I'd say the standard statistics answer is to compare to the intercept only model. (this handles the unbalanced classes mentioned in other answers) cf mcFadden's pseudo r^2. https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/
Now the problem is what the maximum value is. fundamentally the problem is that probability of an event is undefined outside a model for the events. the way I would suggest is that you take your test data and aggregate it to a certain level, to get a probability estimate. then calculate the logloss of this estimate.
eg you are predicting click through rate based on (web_site, ad_id, consumer_id), then you aggregate clicks, impressions to eg web_site level and calculate the ctr on the test set for each web site. then calculate log_loss on your test data_set using these test click through rates as predictions. This is then the optimal logloss on your test set for a model only using website ids. The problem is we can make this loss as small as we like by just adding more features until each record is uniquely identified.
$\begingroup$
$\endgroup$
As others have pointed out, the - log of the loss = (probability of correct classification). So, for example, losses of -log(.9), -log(.8), -log(.7), -log(.6), and -log(.5), or .11, .22, .36, .51, and .69 corresponding to probabilities of correct classification of 90%, 80%, 70%, 60%, 50%. Thinking evaluatively, a random classifier will, on average, make correct predictions in a balanced classification problem 1/n_classes % of the time, so the loss of a random classifier would be -log(1/n_classes). However, by log laws, this is equal to log((1/n_classes)^-1) = log(n_classes). For n_classes on [2,10], random classifiers should produces losses [log(2), log(10)], or (0.69, 1.10, 1.39, 1.61, 1.79, 1.95, 2.08, 2.20, 2.30) respectively if you're looking for some concrete numerical benchmarks. For unbalanced classification refer to Fed Zee's answer, and for determining the significance of a better-than-random log loss look at significance testing with binomial distributions.
