Just wondering the effects of additional features. Following are several thoughts:

  1. If the additional features are noisy (can not distinguish the two classes), then additional features won't hurt SVM. Because SVM's final hyperplane will parallel to the additional dimension.

  2. If the additional features happen to provide some distinguish power on the training data, SVM will use these features to grasp some unreal trend, thus these additional features will increase SVM's generalization error.

  3. If there are a lot of training data, then the probability that a feature happen to distinguish the two classes is low.

  4. If you add too many features, then the probability that these additional features together happen to distinguish the two classes is high.

Am I correct?


With respect to 1, I think that adding uninformative features will impact the classifiers performance. The degree to which the performance is affected depends on the ratio of informative features/ amount training data (point 3 and 4 that you made) and noise in the training data.But generally the SVMs are often said to be robust to noise.

I think, the second point may be true but can be checked by ensuring that the training and test data is chosen appropriately and the error is calculated well. This may not be a problem in SVM, but rather a general problem in all machine learning problems.

  • $\begingroup$ I agree with @lekshmi and want to emphasize that SVMs are affected with feature subset selection where different subsets of features lead to different performance levels. It is not that SVMs explicitly limit effect of any noisy feature. Yet, these models might be less sensitive to such noisy features compared to a classifier like Naive Bayes Classifier. $\endgroup$ – soufanom Apr 10 '15 at 3:05
  • 1
    $\begingroup$ @soufanom, according to this post stats.stackexchange.com/questions/63081/… , the top answer says that we should not do feature selection before SVM, which could lead to overfitting even more. $\endgroup$ – Xing Shi Apr 10 '15 at 20:49
  • $\begingroup$ @EarthWorm thanks for referring me to the post which is very interesting. It might worth looking at the following reference by the authors of SVM themselves: Guyon, Isabelle, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. "Gene selection for cancer classification using support vector machines." Machine learning 46, no. 1-3 (2002): 389-422. $\endgroup$ – soufanom Apr 12 '15 at 8:13
  • $\begingroup$ An interesting quote relevant to your question in the problem description of this paper is: "Training techniques that use regularization (see e.g. Vapnik, 1998) avoid overfitting of the data to some extent without requiring space dimensionality reduction. Such is the case, for instance, of Support Vector Machines (SVMs) (Boser, 1992; Vapnik, 1998; Cristianini, 1999). Yet, as we shall see from experimental results (Section 5), even SVMs benefit from space dimensionality reduction." $\endgroup$ – soufanom Apr 12 '15 at 8:14
  • $\begingroup$ Thus, it is difficult to say that we should "not" do feature selection before SVM. Rather, one may try do so but with a careful experimental setup that shall limit over-fitting effect of feature selection over SVMs or other classifiers. $\endgroup$ – soufanom Apr 12 '15 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.