I am trying to get a grasp on what is the purpose of the loss function and I can't quite understand it.

So, as far as I understand loss function is for introducing some kind of metric that we can measure the "cost" of an incorrect decision with.

So let's say I have a dataset of 30 objects, I divided them to training / testing sets like 20 / 10. I will be using 0-1 loss function, so lets say my set of class labels is M and the function looks like this:

$$ L(i, j) = \begin{cases} 0 \qquad i = j \\ 1 \qquad i \ne j \end{cases} \qquad i,j \in M $$

So I builded some model on my training data, lets say I am using Naive Bayes classifier, and this model classified 7 objects correctly (assigned them the correct class labels) and 3 objects were classified incorrectly.

So my loss function would return "0" 7 times and "1" 3 times - what kind of information can I get from that? That my model classified 30% of the objects incorrectly? Or is there more to it?

If there are any mistakes in my way of thinking I am very sorry, I am just trying to learn. If the example I provided is "too abstract", let me know, I'll try to be more specific. If you will try ti explain the concept using different example, please use 0-1 loss function.


You have correctly summarized the 0-1 loss function as effectively looking at accuracy. Your 1's become indicators for misclassified items, regardless of how they were misclassified. Since you have three 1's out of 10 items, your classification accuracy is 70%.

If you change the weighting on the loss function, this interpretation doesn't apply anymore. For example, in disease classification, it might be more costly to miss a positive case of disease (false negative) than to falsely diagnose disease (false positive). In this case, your loss function would weight false negative misclassification more heavily. The sum of your losses would no longer represent accuracy in this case, but rather the total "cost" of misclassification. The 0-1 loss function is unique in its equivalence to accuracy, since all you care about is whether you got it right or not, and not how the errors are made.

  • $\begingroup$ @JohnnyJohansson that is the definition of accurracy in statistics, see en.wikipedia.org/wiki/Sensitivity_and_specificity $\endgroup$ – Tim Jun 7 '17 at 15:07
  • $\begingroup$ @Tim - I am still confused by the 0-1 loss function - could the resulting matrix have any values greater than 1, i.e. if there are 3 miss classifications we would see a value of 3 in the corresponding entry ? see here math.stackexchange.com/questions/2623072/… $\endgroup$ – Xavier Bourret Sicotte Jun 22 '18 at 12:10
  • 1
    $\begingroup$ @XavierBourretSicotte The cost matrix values don't depend on the number of misclassifications. You could define the matrix so that misclassifying Class A as Class B has a cost of 1, but the reverse has a cost of 3 (or any arbitrary values for either, really, it's more about relative size of the costs). Then you look at your actual predictions, and sum up the total cost of your misclassifications, however many there may be. With 0-1 cost, the total cost is equal to the number of misclassified items, but for an arbitrary cost function, it's an arbitrary-scale score where lower is better. $\endgroup$ – Nuclear Hoagie Jun 28 at 16:40

Yes, this is basically it: you count the number of misclassified items. There is nothing more behind it, it is a very basic loss function. What follows, 0-1 loss leads to estimating mode of the target distribution (as compared to $L_1$ loss for estimating median and $L_2$ loss for estimating mean).


I think your confusion is not differentiating the loss for one data point vs. the loss for the whole data set.

Specifically, your $L(y,\hat y)$ is the loss for one data point (I am changing the notation little bit). And the loss for the whole data set, i.e., classification accuracy, needs to summing all data points.

$$ \sum_i L(y_i,\hat y_i) $$

  • $\begingroup$ I actually get the difference, but it is hard for me to understand what would I need this loss for one data point other than calculating loss for the whole dataset? And what should I consider when choosing adequate loss function for some particular problem? $\endgroup$ – Johnny Johansson Jun 7 '17 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.