Does the class label has an influence on the accuracy?

Question

I recently had a discussion about machine learning. They said that they had problems with "accuracy" of the classifier, as far as I understood they used SVM. To solve the problem, they then switched the class label. The classifier has two classes, class A was labeled with 0 while class B was labeled with 1, so after the switch class A would be 1 and class B 0. But they would not change the definition, that samples with label 1 are "positive" and label 0 corresponds to a "negative" result.

They stated, that this switch alone increased the "accuracy" of the classifier. I write "accuracy", as I'm not sure if they actually meant the metric.

I do not see a reason why switching the class label should have any influence on the classifier. All you should see is, that the confusion matrix flips around, as you changed somehow the definition of "positive" and "negative" classes. Hence, also the metrices like precision and recall change, but the accuracy should stay the same.

But maybe I'm missing something here. Are there any classifiers which will have different results depending on the labeling of the classes?

Answer 1

It should have zero influence.

If your classifier (or its results) depend on whether you label your instances "0" and "1" versus "TRUE" and "FALSE" versus "vanilla" versus "chocolate", then something is very wrong. It may be that they inadvertently included the label as a feature, which defeats the purpose.

Answer 2

The evaluation metric may in fact be dependent on the labels, especially for imbalanced data.

For imbalanced data, the accuracy is typically a misleading metric, since the common interpretation is that 1 is ideal and 0 is bad. However, given that the majority class may already make up 99% of the data, a simple model predicting always the majority class will have an accuracy of 99%. It is true though, as you already stated, that the accuracy is independent from the labels.

However, my guess it that they were faced with an imbalanced data set and thus were talking of a different metric that aims to put more importance on the minority class and it's classification.

Short Reminder: When dealing with (binary) classification data, we typically use the confusion matrix to summarize our results:

Confusion Matrix

One such metric which addresses imbalanced data is recall, also referred to as sensitivity or True Positive Rate (TPR). It aims to summarize how well classification works on the Positive-class. It is defined as follows $$ \text{Recall} = \frac {\text{True Positives (i.e., classified Positive while being Positive)}} {\text{Positives (i.e., number of samples that have a Positive label assigned)}} $$ Clearly, recall is dependent on the labels, or more concretely on the choice which label corresponds to Positive. As stated earlier, in imbalanced data, we aim to put more importance on the minority class, thus one should use the minority class as Positive.

Let us close with an example to get a better understanding of what happens when labels are switched. For completeness, I also add the precision metric which aims to summarize how well positive identifications are, see this.

Correct Definition (Positive = Minority Class)	Misleading Definition (Positive = Majority Class)
$\text{Recall}=\frac {5.000}{5.000 + 4.000} \approx 0.56 $	$\text{Recall} = \frac {50.000} {50.000 + 500} \approx 0.99 $
$\text{Precision}=\frac {5.000}{5.000 + 500} \approx 0.91 $	$\text{Precision} = \frac {50.000} {50.000 + 4.000} \approx 0.93 $