Log Loss function in scikit-learn returns different values

Question

I have been trying to wrap my head around the log loss function for model evaluation. I understand how the value is calculated after doing the math by hand.

In the python module sklearn.metrics the log_loss function returns two different values depending on the order of the input lables.

from sklearn.metrics import log_loss
y_pred = [[ 0.1 ,  0.9 ],
          [ 0.9 ,  0.1 ],
          [ 0.8 ,  0.2 ],
          [ 0.35,  0.65]]

log_loss(["ham", "spam", "spam", "ham"], y_pred)
1.8161075557302173

log_loss(["spam", "ham", "ham", "spam"], y_pred)
0.21616187468057912

What is the correct proceedure? I would have thought that the output for both would be the same. When tackling a real machine learning problem, how will I know which metric is the correct one?

PS. Apologies if this question should have been posted in stack-overflow

Answer 1

Half of the answer is that the predictions are not symetrically 'false'. Thus you would not expect the error to be equal if you invert the true labels.

Having a look at the documentation and from there at the source code one can find the relevant formula after binarization of labels. If I now apply that formula to the binarized labels I will see these results. From here on you will understand the rest yourself.

transformed_labels = np.array([[1],
       [0],
       [0],
       [1]])

y_pred = [[ 0.1 ,  0.9 ],
          [ 0.9 ,  0.1 ],
          [ 0.8 ,  0.2 ],
          [ 0.35,  0.65]]

-(transformed_labels * np.log(y_pred)).sum(axis=1)
array([ 2.40794561, -0.        , -0.        ,  1.48060504])

transformed_labels = np.array([[0],
       [1],
       [1],
       [0]])

-(transformed_labels * np.log(y_pred)).sum(axis=1)
array([-0.        ,  2.40794561,  1.83258146, -0.        ])