Identifiying performance of individual feature components in accuracy of support vector methods

Question

I'm using support vector regression (not classification) for a problem and it's working well.

However, in the older method that former lab members developed (a basic linear model, with weights determined just from OLS), there is a lot of code used to determine the individual efficacy of each different right-hand-side variable from their OLS. I'm not all that impressed with these efficacy stats (they are mostly summary stats of correlations, t-stats, etc.) but they do allow you to have some notion of which RHS variables are most informative.

In my case, for each individual datum, I use the whole row's worth of right-hand-side variables as the feature vector for support vector regression. So I am wondering if there is any commonly used analogue, maybe some measure from information theory or something, that gives an indication of how each marginal component of the overall feature vector contributes to the end performance.

One thing that does not seem to work is training N different regressions each using only a single one of the feature components as the sole predictor. This does give some matches between the univariate regression efficacy and univariate support vector efficacy, but it doesn't capture the interactions between the different predictors, and (for example) which predictor is the "second best" predictor among all the components of the feature vector.

I suppose I could work backward and test the results of removing one component at a time, to see which then causes the biggest drop in performance... but I'd prefer something less ad-hoc and perhaps backed by some statistical rigor or intuition.

Answer 1

There are a number of ways of performing relevance determination / feature selection in general. One common way is using an algorithm called Recursive Feature Elimination (RFE) - it's implemented in scikit-learn. This effectively computes a different objective function to what the SVM normally uses to assess the relevant of a subset of feature and keep discarding features while this objective function is increasing. It's expensive as it means training an SVM a number of times and you don't want to over-fit so these results should also be cross-validated. Here's an example implementation. Another method of feature selection is to use a ANOVA to assess the relative importance of each feature in the training set on the determination of the associated label. This should extend to the case of regression. Here's an explanation that uses a Chi squared test. It's also implemented in scikit-learn if I remember correctly.

EDIT: Found it! Chi Squared Feature selection in scikit-learn.

For an excellent overview of the different strategies of feature selection in general have a look at Guyon's paper. This breaks feature selection down into three classes (wrapper, filter and embedded methods) depending on where in the pipeline it is done. It also discusses the merits of each.

Something worth mentioning is Multiple Kernel Learning (MKL) where you learn a weight across a number of kernels of SVMs. This is more complicated but powerful framework for implicitly solving the feature selection problem. MKL is not implemented in scikit-learn, but it is implemented extensively in the Shogun toolkit. EDIT: Example. There is also an implementation called SimpleMKL in MATLAB. EDIT: Here's the code

MKL allows for a large amount of flexibility in incorporating different kernels and heterogenous datasources as well as performing feature selection at the same time. Down side is that it is slower, but if you are going to use RFE anyway then MKL might be a good choice. Two important papers showing different MKL algorithms are SimpleMKL MATLAB toolbox is also available, and Lp-Norm MKL implemented in Shogun. Ultimately SimpleMKL uses gradient descent to solve the outer problem and is fast. The other method uses cutting planes to lower-bound the objective function and is slow but can tackle large scale data. Recently they've even taken the limit of this process to Infinite Kernel Learning.

Parameter optimisation is also expensive in MKL as there are sometimes too many parameters for a grid search so I would suggest a random search, or wait for my dissertation :)

Aside: If you aren't constrained to SVM's I would also look at Gaussian processes. Specifically with an Automatic Relevant Determination (ARD) kernel. If you optimize the marginal log-likelihood of the model the ARD parameters will give you an understand of the importance of each feature in your data. Check our Rasmussen William textbook free online. The documentation for the included MATLAB toolbox includes an example doing exactly this. Scroll down to: "We specify a Gaussian process model as follows"

Hope that helps.