How to use Weight vector of SVM and logistic regression for feature importance?

Question

I have trained a SVM and logistic regression classifier on my dataset. Both classifier provide a weight vector which is of the size of the number of features. I can use this weight vector to select the 10 most important features by just selecting the 10 features with the highest weights.

Should I use the absolute values of the weights, i.e. selecting the 10 features with the highest absolute values?

Second, this only works for SVM with linear kernel but not with RBF kernel as I have read. For non-linear kernel the weights are somehow no more linear. What is the exact reason that the weight vector cannot be used to determine the importance of features in case of non-linear kernel SVM?

Answer 1

It might depend on the scaling of your features: remember $w$ is, indeed, a geometric hyperplane which normal vector is given by weights on features, and these weights give a sense of importance to features (weights $w_i \approx 0$ mean that feature doesn't contribute much to the hyperplane, the plane being almost constant in it).

About other kernels, you have to understand the separating hyperplane is given not in the feature space, but on the kernelized space, whose dimension might even be infinite (in the case of the RBF kernel, for example).

It's really hard (or even impossible) to transpose the weights on the kernelized space to the feature space due to the non-parametric nature of the solution of SVMs for most kernels (the downside of the dual formulation).