0-1 Loss Function explanation

Question

I am trying to get a grasp on what is the purpose of the loss function and I can't quite understand it.

So, as far as I understand loss function is for introducing some kind of metric that we can measure the "cost" of an incorrect decision with.

So let's say I have a dataset of 30 objects, I divided them to training / testing sets like 20 / 10. I will be using 0-1 loss function, so lets say my set of class labels is M and the function looks like this:

$$ L(i, j) = \begin{cases} 0 \qquad i = j \\ 1 \qquad i \ne j \end{cases} \qquad i,j \in M $$

So I built a model on my training data, say, using Naive Bayes classifier, and this model classified 7 objects correctly (assigned them the correct class labels) and 3 objects incorrectly.

So my loss function would return "0" 7 times and "1" 3 times - what kind of information can I get from that? That my model classified 30% of the objects incorrectly? Or is there more to it?

If there are any mistakes in my way of thinking I am very sorry, I am just trying to learn. If the example I provided is "too abstract", let me know, I'll try to be more specific. If you wish to explain the concept using a different example, please use 0-1 loss function.

Answer 1

You have correctly summarized the 0-1 loss function as effectively looking at accuracy. Your 1's become indicators for misclassified items, regardless of how they were misclassified. Since you have three 1's out of 10 items, your classification accuracy is 70%.

If you change the weighting on the loss function, this interpretation doesn't apply anymore. For example, in disease classification, it might be more costly to miss a positive case of disease (false negative) than to falsely diagnose disease (false positive). In this case, your loss function would weight false negative misclassification more heavily. The sum of your losses would no longer represent accuracy in this case, but rather the total "cost" of misclassification. The 0-1 loss function is unique in its equivalence to accuracy, since all you care about is whether you got it right or not, and not how the errors are made.

Answer 2

Yes, this is basically it: you count the number of misclassified items. There is nothing more behind it, it is a very basic loss function. What follows, 0-1 loss leads to estimating mode of the target distribution (as compared to $L_1$ loss for estimating median and $L_2$ loss for estimating mean).

Answer 3

I think your confusion is not differentiating the loss for one data point vs. the loss for the whole data set.

Specifically, your $L(y,\hat y)$ is the loss for one data point (I am changing the notation little bit). And the loss for the whole data set, i.e., classification accuracy, needs to summing all data points.

$$ \sum_i L(y_i,\hat y_i) $$