There are a number of ways of performing relevance determination / feature selection in general. One common way is using an algorithm called Recursive Feature Elimination (RFE) - it's implemented in scikit-learn. This effectively computes a different objective function to what the SVM normally uses to assess the relevant of a subset of feature and keep discarding features while this objective function is increasing. It's expensive as it means training an SVM a number of times and you don't want to over-fit so these results should also be cross-validated. Here's an example implementation. Another method of feature selection is to use a ANOVA to assess the relative importance of each feature in the training set on the determination of the associated label. This should extend to the case of regression. Here's an explanation that uses a Chi squared test. It's also implemented in scikit-learn if I remember correctly.

EDIT: Found it! Chi Squared Feature selection in scikit-learn.

For an excellent overview of the different strategies of feature selection in general have a look at Guyon's paper. This breaks feature selection down into three classes (wrapper, filter and embedded methods) depending on where in the pipeline it is done. It also discusses the merits of each.

Something worth mentioning is Multiple Kernel Learning (MKL) where you learn a weight across a number of kernels of SVMs. This is more complicated but powerful framework for implicitly solving the feature selection problem. MKL is not implemented in scikit-learn, but it is implemented extensively in the Shogun toolkit. EDIT: Example. There is also an implementation called SimpleMKL in MATLAB. EDIT: Here's the code

I would say the state of the art is MKL. It allows for a large amount of flexibility in incorporating different kernels and heterogenous datasources as well as performing feature selection at the same time. Down side is that it is slower, but if you are going to use RFE anyway then MKL might be a good choice. Two important papers showing different MKL algorithms are SimpleMKL MATLAB toolbox is also available, and Lp-Norm MKL implemented in Shogun. Ultimately SimpleMKL uses gradient descent to solve the outer problem and is fast. The other method uses cutting planes to lower-bound the objective function and is slow but can tackle large scale data. Recently they've even taken the limit of this process to Infinite Kernel Learning.

Parameter optimisation is also expensive in MKL as there are sometimes too many parameters for a grid search so I would suggest a random search, or wait for my dissertation :)

Aside: If you aren't constrained to SVM's I would also look at Gaussian processes. Specifically with an Automatic Relevant Determination (ARD) kernel. If you optimize the marginal log-likelihood of the model the ARD parameters will give you an understand of the importance of each feature in your data. Check our Rasmussen William textbook free online. The documentation for the included MATLAB toolbox includes an example doing exactly this. Scroll down to: "We specify a Gaussian process model as follows"

Hope that helps.

There are a number of ways of performing relevance determination / feature selection in general. One common way is using an algorithm called Recursive Feature Elimination (RFE) - it's implemented in scikit-learn. This effectively computes a different objective function to what the SVM normally uses to assess the relevant of a subset of feature and keep discarding features while this objective function is increasing. It's expensive as it means training an SVM a number of times and you don't want to over-fit so these results should also be cross-validated. Here's an example implementation. Another method of feature selection is to use a ANOVA to assess the relative importance of each feature in the training set on the determination of the associated label. This should extend to the case of regression. Here's an explanation that uses a Chi squared test. It's also implemented in scikit-learn if I remember correctly.

EDIT: Found it! Chi Squared Feature selection in scikit-learn.

For an excellent overview of the different strategies of feature selection in general have a look at Guyon's paper. This breaks feature selection down into three classes (wrapper, filter and embedded methods) depending on where in the pipeline it is done. It also discusses the merits of each.

Something worth mentioning is Multiple Kernel Learning (MKL) where you learn a weight across a number of kernels of SVMs. This is more complicated but powerful framework for implicitly solving the feature selection problem. MKL is not implemented in scikit-learn, but it is implemented extensively in the Shogun toolkit. EDIT: Example. There is also an implementation called SimpleMKL in MATLAB. EDIT: Here's the code

MKL allows for a large amount of flexibility in incorporating different kernels and heterogenous datasources as well as performing feature selection at the same time. Down side is that it is slower, but if you are going to use RFE anyway then MKL might be a good choice. Two important papers showing different MKL algorithms are SimpleMKL MATLAB toolbox is also available, and Lp-Norm MKL implemented in Shogun. Ultimately SimpleMKL uses gradient descent to solve the outer problem and is fast. The other method uses cutting planes to lower-bound the objective function and is slow but can tackle large scale data. Recently they've even taken the limit of this process to Infinite Kernel Learning.

Parameter optimisation is also expensive in MKL as there are sometimes too many parameters for a grid search so I would suggest a random search, or wait for my dissertation :)

Aside: If you aren't constrained to SVM's I would also look at Gaussian processes. Specifically with an Automatic Relevant Determination (ARD) kernel. If you optimize the marginal log-likelihood of the model the ARD parameters will give you an understand of the importance of each feature in your data. Check our Rasmussen William textbook free online. The documentation for the included MATLAB toolbox includes an example doing exactly this. Scroll down to: "We specify a Gaussian process model as follows"

Hope that helps.

There are a number of way of performing relevance determination / feature selection. One common way is using an algorithm called Recursive Feature Elimination (RFE) it's implemented in scikit-learn. This effectively compute a different objective function to what the SVM normally used to assess the relevant of a subset of feature and keep discarding features while this objective function is increasing. It's expensive as it means training SVM a number of times and you don't want to over-fit so these results should also be cross-validated. Here's an example implementation. Another method of feature selection is to use a ANOVA to assess the relative importance of each feature in the training set on the determination of the associated label. This should extend to the case of regression. Here's an explanation that uses a Chi squared test. It's also implemented in scikit-learn if I remember correctly.

For an excellent overview of the different strategies of feature selection in general have a look at Guyon's paper. This break's feature selection down into three classes (wrapper, filter and embedded methods) depending on where in the pipeline it is done.

Something worth mentioning is Multiple Kernel Learning (MKL) where you learn a weight across a number of kernel of SVMs. This is more complicated but powerful framework for implicitly solving the feature selection problem. MKL is not implemented in scikit-learn, but it is implemented extensively in the Shogun toolkit. There is also an implementation called SimpleMKL in MATLAB.

I would say the state of the art is MKL. It allows for a large amount of flexibility in incorporating different kernels and heterogenous datasources as well as performing feature selection at the same time. Down side is that is't slowed, but if you are going to use RFE then MKL might be a good choice. Two important papers showing different MKL algorithms are SimpleMKL MATLAB toolbox is also available, and Lp-Norm MKL implemented in Shogun. Ultimately SimpleMKL uses gradient descent to solve the outer problem and is fast. The other method uses cutting planes to lower-bound the objective function and is slow but can tackle large scale data.

Parameter optimisation is also expensive in MKL as there are sometimes too many parameters for a grid search so I would suggest a random search, or wait for my dissertation :).

Aside: If you aren't constrained to SVM's I would also look at Gaussian processes. Specifically with an Automatic Relevant Determination (ARD) kernel. If you optimize the marginal log-likelihood of the model the ARD parameters will give you an understand of the importance of each feature in your data. Check our Rasmussen William textbook free online. The documentation for the included MATLAB toolbox includes an example doing exactly this. Scroll down to: "We specify a Gaussian process model as follows"

Hope that helps.

There are a number of ways of performing relevance determination / feature selection in general. One common way is using an algorithm called Recursive Feature Elimination (RFE) - it's implemented in scikit-learn. This effectively computes a different objective function to what the SVM normally uses to assess the relevant of a subset of feature and keep discarding features while this objective function is increasing. It's expensive as it means training an SVM a number of times and you don't want to over-fit so these results should also be cross-validated. Here's an example implementation. Another method of feature selection is to use a ANOVA to assess the relative importance of each feature in the training set on the determination of the associated label. This should extend to the case of regression. Here's an explanation that uses a Chi squared test. It's also implemented in scikit-learn if I remember correctly.

EDIT: Found it! Chi Squared Feature selection in scikit-learn.

For an excellent overview of the different strategies of feature selection in general have a look at Guyon's paper. This breaks feature selection down into three classes (wrapper, filter and embedded methods) depending on where in the pipeline it is done. It also discusses the merits of each.

Something worth mentioning is Multiple Kernel Learning (MKL) where you learn a weight across a number of kernels of SVMs. This is more complicated but powerful framework for implicitly solving the feature selection problem. MKL is not implemented in scikit-learn, but it is implemented extensively in the Shogun toolkit. EDIT: Example. There is also an implementation called SimpleMKL in MATLAB. EDIT: Here's the code

I would say the state of the art is MKL. It allows for a large amount of flexibility in incorporating different kernels and heterogenous datasources as well as performing feature selection at the same time. Down side is that it is slower, but if you are going to use RFE anyway then MKL might be a good choice. Two important papers showing different MKL algorithms are SimpleMKL MATLAB toolbox is also available, and Lp-Norm MKL implemented in Shogun. Ultimately SimpleMKL uses gradient descent to solve the outer problem and is fast. The other method uses cutting planes to lower-bound the objective function and is slow but can tackle large scale data. Recently they've even taken the limit of this process to Infinite Kernel Learning.

Parameter optimisation is also expensive in MKL as there are sometimes too many parameters for a grid search so I would suggest a random search, or wait for my dissertation :)

Aside: If you aren't constrained to SVM's I would also look at Gaussian processes. Specifically with an Automatic Relevant Determination (ARD) kernel. If you optimize the marginal log-likelihood of the model the ARD parameters will give you an understand of the importance of each feature in your data. Check our Rasmussen William textbook free online. The documentation for the included MATLAB toolbox includes an example doing exactly this. Scroll down to: "We specify a Gaussian process model as follows"

Hope that helps.