1
$\begingroup$

I trained two svms (LIBSVM) with 15451 samples after I did a 10-fold cross-validation and found the best parameter values for gamma and C (RBF kernel). In one svm I used just 1 feature and in the second an additional one (to see whether this additional is improving prediction). After CV I have am accuracy of 75 % (SVM with one feature) and 77 % (SVM with that additional one). After testing on another 15451 instances I have an accuracy of 70 % and 72 % respectively.

I know that this is called overfitting but is it significant here, since it is only a difference of 5 %.

What could I do to avoid overfitting?

Is it even good to use just one or two features and a relatively big training set?

Improve this question
$\endgroup$
4
  • $\begingroup$ What's the difference between the two samples? Check some summary statistics and compare them. Are they comparable? $\endgroup$
    – wcampbell
    Commented May 9, 2014 at 18:02
  • $\begingroup$ I am not sure what to compare? If you mean the training and testing set, the fraction of neutrals is 50 % (training) and 60 % (testing). When adding this feature in the second SVM, only 14 % of instances in both training and testing set have that feature above (in other words, have that feature at all). $\endgroup$
    – user3585509
    Commented May 9, 2014 at 18:34
  • $\begingroup$ It's not called overfiting, it's called using wrong method for a wrong problem. $\endgroup$
    – rep_ho
    Commented Jan 23, 2016 at 13:35
  • $\begingroup$ Is it a balanced problem? How do you find the values of your parameters? Maybe your features aren't descriptive enough. If you only have two features, try to inspect your data, and see if you gain some insight (boxplots, scatter plot,...) $\endgroup$
    – jpmuc
    Commented Jan 23, 2016 at 19:52

2 Answers 2

Reset to default
1
$\begingroup$

Using only one feature for classification seems arbitrary. The way I see it, there are two possibilities. (i) increase number of features (ii) consider permutation tests. For (i), I can direct you to Fast Feature Selection. Also, I implemented the procedure for MATLAB and LIBSVM for binary classification. Code is found at github. For (ii), permutation tests require multiple runs (e.g. 100, 1000) of classifier evaluation where labels are randomly permuted. In the end accuracies are sorted by runs to obtain the significance level at p=0.05. Assume you did 100 permutation runs, then you obtain the significance level at position 5 or 95 depending on the ordering.

Improve this answer
$\endgroup$
1
$\begingroup$

You increased the sample size by quite a bit to observe the 5% drop in generalization performance. One reason for this is that you used cross-validation to hypertune your model, but cross-validation error is not an unbiased estimator for true prediction error. So this type of behavior is not unusual. One thing you may consider to get a "better" (lower variance or maybe lower bias, depending on what you do) estimator for prediction error:

  1. Split your data into a training, validation, and prediction set; hypertune with the validation set and get a realistic performance metric from prediction set
  2. Use a different number of folds (like leave-one-out or five-fold cross-validation)
  3. Try something that is proven to be unbiased like the .632+ rule
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.