1
$\begingroup$

I have a dataset for classification (binary - 1/0) that has around 4000 samples that I use to train the model (I'm using an SVM, if that's relevant). To check whether everything is working fine, I cross validate 5 folds. But I'm noticing that the cross validation accuracy that I get is the exact same fraction of the 0 labels in the set.

For instance, when I pick the last 800 samples for testing and train on the first 3200 samples, the accuracy of prediction on the last 800 samples is the exact same number as the fraction of samples having label = 0 in those 800 data points. Does this mean that the classifier is not taking the features into consideration at all? Or is this completely normal behavior and I'm missing something?

| cite | improve this question | | | | |
$\endgroup$
  • 1
    $\begingroup$ So basically what you're saying is that your SVM classifier is classifying all testing data-set as 0. Is your data-set balanced (#of 1s is approximately equals to #of 0s)? $\endgroup$ – Roronoa Zoro Sep 2 '12 at 20:28
  • $\begingroup$ @RoronoaZoro I wouldn't call it balanced. Around 70% are 1s and the rest 0s. $\endgroup$ – Siddhant Sep 3 '12 at 19:40
  • $\begingroup$ That's odd. I would imagine it's the other way around. So you're accuracy (# correctly classified / # all instances) would be really low then... Since you have more 1s... The bad thing about SVM is that there are many parameters to tweak, if you're using Libsvm library, you should play with the parameter -w (the weight)... Basically you increase the weight/penalty of classifying 1s as 0s. $\endgroup$ – Roronoa Zoro Sep 3 '12 at 20:04
2
$\begingroup$

This sounds as though your classifier is assigning all cases to zero (classifying all cases into class labelled by 0). I suggest calculating the sensitivity for each class, i.e. the proportion of each class correctly classified, in order to check if this is the case. If you have unbalanced classes, try minimizing something other than overall classification error, such as (1-ROC area) or mean classification error across classes.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.