LIBSVM overfitting I trained two svms (LIBSVM) with 15451 samples after I did a 10-fold cross-validation and found the best parameter values for gamma and C (RBF kernel). In one svm I used just 1 feature and in the second an additional one (to see whether this additional is improving prediction). After CV I have am accuracy of 75 % (SVM with one feature) and 77 % (SVM with that additional one). After testing on another 15451 instances I have an accuracy of 70 % and 72 % respectively.
I know that this is called overfitting but is it significant here, since it is only a difference of 5 %.
What could I do to avoid overfitting?
Is it even good to use just one or two features and a relatively big training set?    
 A: Using only one feature for classification seems arbitrary. The way I see it, there are two possibilities. (i) increase number of features (ii) consider permutation tests.
For (i), I can direct you to Fast Feature Selection. Also, I implemented the procedure for MATLAB and LIBSVM for binary classification. Code is found at github.
For (ii), permutation tests require multiple runs (e.g. 100, 1000) of classifier evaluation where labels are randomly permuted. In the end accuracies are sorted by runs to obtain the significance level at p=0.05. Assume you did 100 permutation runs, then you obtain the significance level at position 5 or 95 depending on the ordering.
A: You increased the sample size by quite a bit to observe the 5% drop in generalization performance. One reason for this is that you used cross-validation to hypertune your model, but cross-validation error is not an unbiased estimator for true prediction error. So this type of behavior is not unusual. One thing you may consider to get a "better" (lower variance or maybe lower bias, depending on what you do) estimator for prediction error:


*

*Split your data into a training, validation, and prediction set; hypertune with the validation set and get a realistic performance metric from prediction set

*Use a different number of folds (like leave-one-out or five-fold cross-validation)

*Try something that is proven to be unbiased like the .632+ rule
