1
$\begingroup$

Using sklearn, I did both a linear kernel SVM and a rbf one. While the rbf gave really great results, I can't determine the important features that the algorithm kept (or used more).

I know that "coef_" does only work for a linear kernel, since for rbf the data space is no longer finite (or at least, it changes [I think]).

Is there a way to actually determine the important features for an RBF kernel SVM ?

Thanks,

$\endgroup$
  • $\begingroup$ Welcome to Cross Validated! Please take a moment to view our tour. It would be helpful if you would tag your question with the software you are using. It is also a best practice to type out your Three Letter Acronyms (TLA) at the first use in your question. $\endgroup$ – Tavrock Mar 6 '17 at 7:07
2
$\begingroup$

Unfortunately not. Although SVMs are often interpreted as transforming your features into a high-dimensional space and fitting a linear classifier in the new space, the transformation is implicit and cannot be easily retrieved.

In fact, SVMs with the RBF kernel behave more like soft nearest neighbours. To see this, denote by $\{x_i,y_i\}_{i=1}^N$ the training data, and $K(.,.)$ the kernel of choice: in this case $K(x,x')=\exp(-\gamma\|x-x'\|^2)$. Then the SVM prediction for an example $x$ takes the form $$ \mathrm{sign}\left( \sum_{i=1}^N \alpha_i y_i K(x_i, x) + \rho \right), $$ where $\rho$ is the intercept_ and $\alpha_i$ are the dual_coef_ in sklearn (see here). As you can see, the decision function is just a linear combination of training labels $y_i$, where the influence of each training example $x_i$ is determined by its overall importance $\alpha_i$ and its distance from $x$, as given by $K$. Closer points have an exponentially larger effect on the prediction, hence the nearest neighbour analogy.

Coming back to your question: maybe it doesn't make so much sense to think in terms of features for SVMs with RBFs (would you ask about features for nearest neighbours?). You may want to have a look at the most influential data points, though, to get a "template" interpretation of your model.

$\endgroup$
  • $\begingroup$ Nice answer (+1) maybe you can add that the rbf kernel implicitly transforms your features into a higher dimensional space of ' transformed features' and tries to separate linearly in that new space. As the transformation is implicit you do not know the link with your original features. $\endgroup$ – user83346 Mar 6 '17 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.