7
$\begingroup$

In several kaggle competitions the scoring was based on "logloss". This relates to classification error.

Here is a technical answer but I am looking for an intuitive answer. I really liked the answers to this question about Mahalanobis distance, but PCA is not logloss.

I can use the value that my classification software puts out, but I don't really understand it. Why do we use it instead of true/false positive/negative rates? Can you help me so that I can explain this to my grandmother or a newbie in the field?

I also like and agree with the quote:

you do not really understand something unless you can explain it to your grandmother
-- Albert Einstein

I tried answering this on my own before posting here.

Links that I did not find intuitive or really helpful include:

These are informative, and accurate. They are meant for a technical audience. They do not draw a simple picture, or give a simple and accessible examples. They are not written for my grandmother.

$\endgroup$
6
$\begingroup$

Logloss is the logarithm of the product of all probabilities. Suppose Alice predicted:

  • with probability 0.2, John will kill Jack
  • with probability 0.001, Mary will marry John
  • with probability 0.01, Bill is a murderer.

It turned out that Mary did not marry John, Bill is not a murderer, but John killed Jack. The product of the probabilities, according to Alice, is 0.2*0.999*0.99=0.197802

Bob predicted:

  • with probability 0.5, John will kill Jack
  • with probability 0.5, Mary will marry John
  • with probability 0.5, Bill is a murderer.

The product is 0.5*0.5*0.5=0.125.

Alice is better predictor than Bob.

$\endgroup$
  • $\begingroup$ why does "product of all probabilities" work? This sounds like a relative of expectation maximization. $\endgroup$ – EngrStudent Apr 20 '16 at 21:23
  • 3
    $\begingroup$ Do you need a formal proof? It is in the "technical answer" mentioned by the topicstarter. Do you need an informal "grandmother" reason why? You say: suppose this fellow gave correct predictions. What is the probability that everything happen as it really happened? This is the product of probabilities. $\endgroup$ – user31264 Apr 20 '16 at 22:55
  • $\begingroup$ "product of probabilities" isn't "grandma". log of product of probabilities is sum of log-probabilities, which they use in expectation maximization and call "expectation". I think it is also encoded in K-L divergence. ... I think in grandma-talk you could say" "most likely" = highest overall probability of multiple events. There are two get "highest": 1) maximize the combined probability or 2) minimize the negative combined probability. Most machine learning likes "gradient descent" or minimizing badness. Log-loss is the negative probability scaled by sample size, and it gets minimized. $\endgroup$ – EngrStudent Jun 30 '17 at 13:11
  • $\begingroup$ Here link they say "exp(-loss) is average probability of correct prediction." $\endgroup$ – EngrStudent Jun 30 '17 at 13:16
  • $\begingroup$ I liked the Bishop ref here. It is equation 4.108 and is the cross-entropy error function. $\endgroup$ – EngrStudent Dec 8 '17 at 16:22

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.