5
$\begingroup$

I have been trying to wrap my head around the log loss function for model evaluation. I understand how the value is calculated after doing the math by hand.

In the python module sklearn.metrics the log_loss function returns two different values depending on the order of the input lables.

from sklearn.metrics import log_loss
y_pred = [[ 0.1 ,  0.9 ],
          [ 0.9 ,  0.1 ],
          [ 0.8 ,  0.2 ],
          [ 0.35,  0.65]]

log_loss(["ham", "spam", "spam", "ham"], y_pred)
1.8161075557302173

log_loss(["spam", "ham", "ham", "spam"], y_pred)
0.21616187468057912

What is the correct proceedure? I would have thought that the output for both would be the same. When tackling a real machine learning problem, how will I know which metric is the correct one?

PS. Apologies if this question should have been posted in stack-overflow

$\endgroup$
1
  • 1
    $\begingroup$ I think this question fits great here since the problem regards the interpretation and understanding of log loss as opposed to programming issues (which do not exist here, since the code works fine) $\endgroup$ Jan 3 '18 at 12:06
2
$\begingroup$

Half of the answer is that the predictions are not symetrically 'false'. Thus you would not expect the error to be equal if you invert the true labels.

Having a look at the documentation and from there at the source code one can find the relevant formula after binarization of labels. If I now apply that formula to the binarized labels I will see these results. From here on you will understand the rest yourself.

transformed_labels = np.array([[1],
       [0],
       [0],
       [1]])

y_pred = [[ 0.1 ,  0.9 ],
          [ 0.9 ,  0.1 ],
          [ 0.8 ,  0.2 ],
          [ 0.35,  0.65]]

-(transformed_labels * np.log(y_pred)).sum(axis=1)
array([ 2.40794561, -0.        , -0.        ,  1.48060504])

transformed_labels = np.array([[0],
       [1],
       [1],
       [0]])

-(transformed_labels * np.log(y_pred)).sum(axis=1)
array([-0.        ,  2.40794561,  1.83258146, -0.        ])
$\endgroup$
3
  • $\begingroup$ Shouldn't both labels be interpreted as [0, 1, 1, 0]? log_loss is receiving the labels but converting them to two different binary encodings. I would've thought the first label encountered would be classed as 0, and the second as 1. $\endgroup$ Jan 3 '18 at 12:19
  • 1
    $\begingroup$ That is an excellent example of an unjustified assumption. If you read the source code you will find that they use scikit-learn.org/stable/modules/generated/…. Then again have a look at this function and test it. $\endgroup$ Jan 3 '18 at 12:27
  • 1
    $\begingroup$ My confusion came from how scikit learn handles encodings. It must use hashes to encode the labels. The ordering of hashed labels would the same regardless of the first samples label. To be specific, if the class of the first sample in the test set differs from that in the training set, the encoding of the classes remains the same, so the log loss of the test set and the train set will be comparable. Thanks for the help. $\endgroup$ Jan 3 '18 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.