I know that Support Vector Machines are very well suited for high-dimensional data, and I have read that one reason is that they have an "inbuilt feature selection". I assume this is due to the fact that Support Vectors are used to build the decision boundary, and other weights could be regularized, i.e. set close to zero. Is this correct, and if so does anyone have this formally or helpful sources?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ "I have read that one reason is that they have an "inbuilt feature selection"." - what is the source? $\endgroup$– Jakub BartczukCommented Mar 27, 2018 at 18:35
-
$\begingroup$ There are at least two ways of regularizing SVMs: soft margin and weight decay. Maybe this wikipedia article will help. $\endgroup$– Jakub BartczukCommented Mar 27, 2018 at 18:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
I think you are on the right track. The essence of it is that an SVM adds dimensions and places the data in that new space - possibly infinite in number of dimensions - and looks for the hyperplane that best separates. In practice, some vectors get lightly weighted. I view it as 'inbuilt' in that - unlike a neural net - it doesn't have to hunt hoping to find a solution; there either is such a hyperplane or there isn't.
However, how and what happens depends a great deal on the kernel you choose and how it is tuned.
A great lecture is here: https://www.youtube.com/watch?v=eHsErlPJWUU
-
$\begingroup$ I think you mean the concept of Kernels, which I understand. So the regularization does not have anything to do with the Support Vectors? $\endgroup$– user24544Commented Mar 27, 2018 at 8:23
-
$\begingroup$ Can you be more specific about what you mean by "regularization"? I assume you mean approaches to avoid overfitting. SVMs work a bit differently than other machine learning algos in that there is an actual optimization, guaranteeing a (globally best) solution if there is one. That is why I referred you to that lecture. This might help: en.wikipedia.org/wiki/… $\endgroup$ Commented Mar 29, 2018 at 3:39