# Test error correlation with label frequency

For a university project I wrote a SVM classifier and during the benchmark phase I'm getting some results I'd like to better understand from the theoretical side.

My original dataset contains a multi-labeled examples but I'm buiding a binary classifier so I pick a feature A and I replace all the labels with +1 if A is present or -1 if it isn't. For instance:

label1 label2 label3 label4 = feature1 feature2 feature3 feature4
label2 label4  = feature1 feature2 feature3 feature4
label3 label4  = feature1 feature2 feature3 feature4


If choose label3 the dataset becomes:

 1 = feature1 feature2 feature3 feature4
-1  = feature1 feature2 feature3 feature4
1  = feature1 feature2 feature3 feature4


If choose label2 the dataset becomes:

 1 = feature1 feature2 feature3 feature4
1  = feature1 feature2 feature3 feature4
-1  = feature1 feature2 feature3 feature4


Now, my original dataset contains 20k rows and:

1. If I choose a label which is in 9k rows I get a average test error of 13%
2. If I choose a label which is in 5k rows I get a average test error of 7%

I was wondering why this is happening. My guesses are:

• In the second case the separating hyperplane has a larger margin
• In this the first case the is overfitting

Be aware that test error rate alone does not give you the full story and may be deceiving. You did not describe the test set, so I am going to assume its distribution is identical to what you use for training (as is the case when you use a random split).

To illustrate why error rate alone is deceiving, we can use a dumb classifier that always yields positive. Lets test it on two different test sets, which is similar to what I believe you are doing:

• balanced test set ($50\%$ positives, $50\%$ negatives) $\rightarrow$ $50\%$ error rate
• unbalanced test set ($90\%$ positives, $10\%$ negatives) $\rightarrow$ $10\%$ error rate

Using a single measure to assess performance is a bad idea. You should look at error rate + NPV + PPV or error rate + sensitivity + specificity to obtain more insight into what is going on.

If you really want to evaluate performance with a single number, some more sensible summaries include area under the ROC or PR curves, for example.

• Thanks for your explanation. BTW, the error I wrote above is the average I got performing k-fold validation (with k=10).
– CwNd
May 29, 2013 at 12:13