Cross validation accuracy is the same as the fraction of negative labels - what does it mean?

Question

I have a dataset for classification (binary - 1/0) that has around 4000 samples that I use to train the model (I'm using an SVM, if that's relevant). To check whether everything is working fine, I cross validate 5 folds. But I'm noticing that the cross validation accuracy that I get is the exact same fraction of the 0 labels in the set.

For instance, when I pick the last 800 samples for testing and train on the first 3200 samples, the accuracy of prediction on the last 800 samples is the exact same number as the fraction of samples having label = 0 in those 800 data points. Does this mean that the classifier is not taking the features into consideration at all? Or is this completely normal behavior and I'm missing something?

Answer 1

This sounds as though your classifier is assigning all cases to zero (classifying all cases into class labelled by 0). I suggest calculating the sensitivity for each class, i.e. the proportion of each class correctly classified, in order to check if this is the case. If you have unbalanced classes, try minimizing something other than overall classification error, such as (1-ROC area) or mean classification error across classes.