I know that Support Vector Machines are very well suited for high-dimensional data, and I have read that one reason is that they have an "inbuilt feature selection". I assume this is due to the fact that Support Vectors are used to build the decision boundary, and other weights could be regularized, i.e. set close to zero. Is this correct, and if so does anyone have this formally or helpful sources?

  • $\begingroup$ "I have read that one reason is that they have an "inbuilt feature selection"." - what is the source? $\endgroup$ Commented Mar 27, 2018 at 18:35
  • $\begingroup$ There are at least two ways of regularizing SVMs: soft margin and weight decay. Maybe this wikipedia article will help. $\endgroup$ Commented Mar 27, 2018 at 18:36

1 Answer 1


I think you are on the right track. The essence of it is that an SVM adds dimensions and places the data in that new space - possibly infinite in number of dimensions - and looks for the hyperplane that best separates. In practice, some vectors get lightly weighted. I view it as 'inbuilt' in that - unlike a neural net - it doesn't have to hunt hoping to find a solution; there either is such a hyperplane or there isn't.

However, how and what happens depends a great deal on the kernel you choose and how it is tuned.

A great lecture is here: https://www.youtube.com/watch?v=eHsErlPJWUU

  • $\begingroup$ I think you mean the concept of Kernels, which I understand. So the regularization does not have anything to do with the Support Vectors? $\endgroup$
    – user24544
    Commented Mar 27, 2018 at 8:23
  • $\begingroup$ Can you be more specific about what you mean by "regularization"? I assume you mean approaches to avoid overfitting. SVMs work a bit differently than other machine learning algos in that there is an actual optimization, guaranteeing a (globally best) solution if there is one. That is why I referred you to that lecture. This might help: en.wikipedia.org/wiki/… $\endgroup$
    – eSurfsnake
    Commented Mar 29, 2018 at 3:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.