3
$\begingroup$

I've tuned a SVM with radial kernel that has training error about 10%, but test error is about 38%, which surprise me.

I tried to understand what may cause this and noticed the number of support vectors is high, about 50% of the observation are support vectors. Is this the cause of the problem? Does high number of support vectors leads to a high variance SVM?

Here is the summary of the SVM. 

Parameters:

   SVM-Type:  C-classification 

 SVM-Kernel:  radial 

       cost:  31.62 

      gamma:  1e-04 

Number of Support Vectors:  117

 ( 58 59 )

Number of Classes:  2 

Levels: 
 high low
Improve this question
$\endgroup$
8
  • $\begingroup$ Test error will always be larger than training error. Why are you surprised, precisely? $\endgroup$
    – Sycorax ♦
    Aug 31, 2015 at 15:13
  • $\begingroup$ @user777 by the difference between training and test error, that is about 28%. I've applied the SVM to other dataset, the differences are usually about 5-10%. $\endgroup$
    – yi.tang.uni
    Aug 31, 2015 at 15:18
  • $\begingroup$ But this is a different data set, correct? Perhaps the relationship between features and outcome is weaker in this data, or there are many low-quality input features, or you need a more specialized kernel function, or you need more data to learn the boundary in the feature space... $\endgroup$
    – Sycorax ♦
    Aug 31, 2015 at 15:20
  • $\begingroup$ How many observations did you have in the training sample ? $\endgroup$
    – user83346
    Aug 31, 2015 at 15:23
  • $\begingroup$ @user777 that's true! I randomly split the observation set into training and test set with 50/50 split. I guess I didn't the expect the relationships between x and y are so different among them. $\endgroup$
    – yi.tang.uni
    Aug 31, 2015 at 15:26

1 Answer 1

Reset to default
3
$\begingroup$

YES, a large number of support vectors is often a sign of overfitting.

The problem appears to be that you have chosen optimal hyperparameters based on training set performance, rather than independent test set performance (or, alternatively, cross-validated estimates).

The problem

Never evaluate a set of hyperparameters based on training set performance, because you will inevitably end up with an immense overfit. Consider the RBF kernel: $$K(\mathbf{u},\mathbf{v}) = \exp(-\gamma \|\mathbf{u}-\mathbf{v}\|^2).$$ Now, take the following limit case: $$ \lim_{\gamma\rightarrow\infty} K(\mathbf{u},\mathbf{v}) = \left\{ \begin{matrix} 0, & \text{if } \mathbf{u}\neq\mathbf{v},\\ 1, & \text{otherwise.} \end{matrix} \right. $$ Which essentially means that, for $\gamma\rightarrow\infty$ the kernel matrix becomes the unit matrix. This allows the resulting model to always fit the training data perfectly, with each instance becoming a support vector, but learn absolutely nothing.

The solution

If you want to optimize hyperparameters, you must always estimate the generalization performance of a given set of hyperparameters based on unseen data. The most common approach is via cross-validation, e.g. $k$-fold or leave-one-out.

Note that, when you are tuning hyperparameters based on cross-validation, it is still effectively possible to overfit. You can reduce this to some extent by using several iterations and/or more folds, but the inherent issue remains.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, thanks for your comments, I really like the example you give to demo the problem. I used the turn function in R to find the optimal parameter, and I believe it use 10-folder CV by default. The high test error rate is probably due to the splitting that gives different data characteristics between the two datasets because the sample size is small (about 400). Thanks for you answer to the question, I'd keep it in mind when apply SVM. $\endgroup$
    – yi.tang.uni
    Sep 10, 2015 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.