I know that log-loss penalises models that are confident with the wrong predicted classes. Can this be translated to percentage accuracy? If not, then how do I report the error or compare it to other percentage error metrics?

For example, on training a neural network with 128 output layers with sigmoid activation, I get a loss reduction from 0.30 to 0.04 over 20 epochs. How do I evaluate classifier accuracy based on this? It is a multi-label classification problem.

  • $\begingroup$ You don't, proper scoring rules don't relate to accuracy at all. There's a hint if you reach the numerical bottom or top of the scale though, but that's it. $\endgroup$ – Firebug May 29 '16 at 21:28
  • $\begingroup$ I read somewhere that it can be interpreted as e^(-total_log_loss). This gives an accuracy estimate (compared to a random guessing model with e^(-log(no_of_labels)). Is this correct or a useful interpretation? $\endgroup$ – goelakash May 29 '16 at 22:06
  • 1
    $\begingroup$ $e^{-\log{K}}=K^{-1}$ is the uniform random guess probability, but that only translates as an accuracy estimate if you assume a naive threshold for classification. On the other hand we have logloss $L = -\sum{\log{P_{i}}}$ and so $e^{-L}=e^{\sum{\log{P_{i}}}}=\prod{P_{i}}$, it takes a single $P_j=0$ to make it reach zero, even if you have all other $P_{i, i \neq j}=1$, and so that cannot be an accuracy estimate. $\endgroup$ – Firebug May 30 '16 at 14:39
  • $\begingroup$ Hmm, that makes sense. But then how is cross-entropy useful at all, if not as a proper metric? If it can't be reported, or compared, how does one define the viability of the model? (Lets say in terms of accuracy/ error rate of classification) $\endgroup$ – goelakash May 30 '16 at 16:14
  • $\begingroup$ It's useful because it penalizes too confident mistakes. It's actually one of the few proper metrics. It also can be reported, if you bound your probability outputs to say $[10^{-15}, 1-10^{-15}]$. Now this part is my opinion: accuracy/error rates are an unnecessary part of the data analysis, dichotomizing your scores/probabilities should be left to a decision-maker. In general, you want the best probabilities estimates possible, and logloss indicates exactly that. $\endgroup$ – Firebug Jun 13 '16 at 23:18

Convert Probability to LogLoss then back to Probablity

var probability = 0.5;
var logloss = Math.Log(probability);
Console.WriteLine(logloss); //-0.693147180559945
var originaProbability = Math.Exp(logloss);
Console.WriteLine(originaProbability); //0.5!

I think this is all it is

  • 2
    $\begingroup$ Could you explain how this might result in the "percentage accuracy" requested in the question? $\endgroup$ – whuber May 8 '19 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.