1
$\begingroup$

I train a SVM classifier using 36 features. If I use all the features, the train accuracy is about 0.96, the test accuracy is about 0.77. Then I change the number of features. The train accuracy drops while the test accuracy remains the same. Even if I just use one feature, the test accuracy is still about 0.77. I use cross validation, but the result does not change. I can not figure it out. So how does it come about?

$\endgroup$
  • $\begingroup$ What's the ratio of positives in your dataset? (Is it 0.77, or 0.23?) $\endgroup$ – Dougal Apr 5 '16 at 2:17
  • $\begingroup$ pages.cs.wisc.edu/~shavlik/roc.pdf $\endgroup$ – Reinstate Monica Apr 5 '16 at 2:22
  • $\begingroup$ @Dougal About 0.77. Is it because that I have an unbalanced training set? Meanwhile, the features are often sparse, it means that maybe many entries of the feature vector are just zero due to the unavailability of data. $\endgroup$ – henrykuo Apr 5 '16 at 2:30
1
$\begingroup$

It seems a bit suspicious that your constant accuracy number and the ratio of positives in your dataset are about the same, doesn't it?

It seems that your model is always predicting a positive label on the test set, though in some cases it's predicting negative on the train set.

Some possible remedies:

  • Try other hyperparameters. This sounds in particular like maybe your RBF kernel bandwidth is too small, if you're using that. Maybe try different kernels altogether.

  • Weight your training instances, so that the model cares more about getting the negatives right than it does about positives. Do this only if that's what you actually care about....

  • Maybe your features just aren't any good. Try getting better ones if you can, or perhaps weighting them differently or otherwise transforming them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.