18
$\begingroup$

I'm trying to better understand log loss and how it works but one thing I can't seem to find is putting the log loss number into some sort of context. If my model has a log loss of 0.5, is that good? What's considered a good and bad score? How do these thresholds change?

$\endgroup$
21
$\begingroup$

The logloss is simply $L(p_i)=-\log(p_i)$ where $p$ is simply the probability attributed to the real class.

So $L(p)=0$ is good, we attributed the probability $1$ to the right class, while $L(p)=+\infty$ is bad, because we attributed the probability $0$ to the actual class.

So, answering your question, $L(p)=0.5$ means, on average, you attributed to the right class the probability $p\approx0.61$ across samples.

Now, deciding if this is good enough is actually application-dependent, and so it's up to the argument.

| cite | improve this answer | |
$\endgroup$
15
$\begingroup$

Like any metric, a good metric is the one better that the "dumb", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.

This "dumb"-guess depends on 2 factors :

  1. the number of classes
  2. the balance of classes : their prevalence in the observed dataset

In the case of the LogLoss metric, one usual "well-known" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.

I. Impact of the number of classes N on the dumb-logloss:

In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :

Logloss = -log(1 / N)

log being Ln, neperian logarithm for those who use that convention.

In the binary case, N = 2 : Logloss = - log(1/2) = 0.693

So the dumb-Loglosses are the following :

dumb-logloss depending of number of classes - balanced case

II. Impact of the prevalence of classes on the dumb-Logloss:

a. Binary classification case

In this case, we predict always p(i) = prevalence(i), and we obtain the following table :

binary classification logloss, balanced - table

So, when classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use

binary classification logloss, unbalanced - chart

b. Three-class case

"Dumb"-logloss depending on prevalence - three-class case :

three-class classification logloss, unbalanced

We can see here the values of balanced binary and three-class cases (0.69 and 1.1).

CONCLUSION

A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.

Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.

In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

So this is actually more complicated than Firebugs response and it all depends on the inherent variation of the process you are trying to predict.

When I say variation what I mean is 'if an event was to repeat under the exact same conditions, known and unknown, what's the probability that the same outcome will occur again'.

A perfect predictor would have a loss, for probability P: Loss = P ln P + (1-P) ln (1-P)

If you are trying to predict something where, at its worse, some events will be predicted with an outcome of 50/50, then by integrating and taking the average the average loss would be: L=0.5

If what you are trying to predict is a tad more repeatable the loss of a perfect model is lower. So for example, say with sufficient information a perfect model was able to predict an outcome of an event where across all possible events the worst it could say is 'this event will happen with 90% probability' then the average loss would be L=0.18.

There is also a difference if the distribution of probabilities is not uniform.

So in answer to your question the answer is 'it depends on the nature of what you are trying to predict'

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It's well known a $L\approx 0.693$ is the non-informative binary logarithmic loss (i.e. random guessing). I don't really follow your calculations, are you assuming an uniform distribution for $p$ or something like that? $\endgroup$ – Firebug Aug 14 '17 at 11:48
  • $\begingroup$ Imagine you have a system, where you know all the possible information about its current state. Imagine that system has some internal level of randomness so that, given all the parameters about its state, an outcome can be different. Say for example in this system it can range from 0-10%. A perfect model (ie one where its performance is only limited by the inherent variation) would get $L\approx 0.18$. For comparison a system which varies from 0-100% would get at its best $L\approx 0.5$. $\endgroup$ – simeon Aug 14 '17 at 12:05
  • 1
    $\begingroup$ I don't disagree with your worse case. I am merely saying that a 'good result' depends on the system. 0.4 may indicate a good result for some systems (like the latter example) or a bad one (for the former). $\endgroup$ – simeon Aug 14 '17 at 12:07
0
$\begingroup$

I'd say the standard statistics answer is to compare to the intercept only model. (this handles the unbalanced classes mentioned in other answers) cf mcFadden's pseudo r^2. https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/

Now the problem is what the maximum value is. fundamentally the problem is that probability of an event is undefined outside a model for the events. the way I would suggest is that you take your test data and aggregate it to a certain level, to get a probability estimate. then calculate the logloss of this estimate.

eg you are predicting click through rate based on (web_site, ad_id, consumer_id), then you aggregate clicks, impressions to eg web_site level and calculate the ctr on the test set for each web site. then calculate log_loss on your test data_set using these test click through rates as predictions. This is then the optimal logloss on your test set for a model only using website ids. The problem is we can make this loss as small as we like by just adding more features until each record is uniquely identified.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.