What machine learning algorithms are good for estimating which features are more important?

Question

I have data with a minimum number of features that don't change, and a few additional features that can change and have a big impact on the outcome. My data-set looks like this:

Features are A, B, C (always present), and D, E, F, G, H (sometimes present)

A = 10, B = 10, C = 10                  outcome = 10
A = 8,  B = 7,  C = 8                   outcome = 8.5
A = 10, B = 5,  C = 11, D = 15          outcome = 178
A = 10, B = 10, C = 10, E = 10, G = 18  outcome = 19
A = 10, B = 8,  C = 9,  E = 8,  F = 4   outcome = 250
A = 10, B = 11, C = 13, E = 8,  F = 4   outcome = 320
...

I want to predict the outcome value, and the combination of additional parameters is very important for determining the outcome. In this example, the presence of E and F leads to a big outcome, whereas the presence of E and G doesn't. What machine learning algorithms or techniques are good to capture this phenomenon ?

Answer 1

This is one of the main areas of research in Machine Learning and it is known as Feature Selection.

In general, the only way to say what the best subset of features is (for input into some predictive model that can combine them), is to try all possible subsets. This is usually impossible, so people try to sample the space of feature subsets by various heuristics (see the article for some typical approaches).

Answer 2

From my understanding, you are looking for a measure of variable importance. These come in several flavours based on several different theoretical approaches, but all have strong links to the methods used to optimise the algorithm you're talking abut. Generally, each machine learning algorithm will have a canonical optimisation method; back propagation for neural networks, sequential minimum optimisation for SVMs, various information criterion and statistical significance tests for decision trees including chi squared significance or gini impurity. Of course, other more novel optimisation methods are frequently proposed for each of the algorithms.

These optimisation methods for each algorithm essentially define the variable importance for the model at hand. Essentially, you're looking for an approximation or interpretable representation of the results of that optimisation step that the algorithm is undertaking. However, this is problematic for several reasons.

1. The difficulty of determining the influence of a given variable on model form selection, given that selection is often a stochastic process itself.The variables influence model selection to some degree, so that even if a variable is not important for the final prediction in a model, it may have crucially shaped the model form itself. Given that the generation of the model itself is often stochastic (optimised using particle swarm optimisation or a bagging method etc.), it is hard to understand exactly how a given variable may have shaped its form.

2. The difficulty of extracting the importance of a single variable given that it may be important only in conjunction or interaction with another variable.

3. Some variables may only be important for some observations. Lack of importance on other observations may confound measurement of overall importance by averaging out a real difference.

It is also hard to get an immediately interpretable metric for variable importance exactly as defined by the model, as it may not produce a single number (especially in the case of bagging). Instead, in these cases there is a distribution of importance for each variable.

One way to overcome these issues might be to use perturbation. This is a way to analyse your final model by adding random noise to your variables, and then checking how this affects the results. The advantage is that it allows you to find which variables are most important empirically through simulation - answering the question of which variables would destroy the prediction most if removed. The disadvantage, is that there is a good chance that even if the variables were removed/perturbed, the model (if re-trained) could use the other variables reconstruct their effect, meaning that the "variable importance" measure you derive still only truly indicates the importance in your trained model, but not the overall importance across all possible models.

Answer 3

As @Bitwise mentions, feature selection or feature extraction is a huge area of research in itself and there are countless ways to do it.

The other answers are all valid in my opinion, but in the end, you will probably do like most, and pick the method that is the most intuitive for you and that you understand the best. I would still add two possible options.

Multiple regression is probably the oldest technique. The idea is to fit a model to describe the response from the predictors and keep only the predictors that have a large impact on the response (a large coefficient of proportionality). Here you would probably have to recode the absence of D, E, F and G as D=0, E=0, F=0, G=0 or something like that.

Another technique that never gained the popularity that it deserves is co-inertia analysis (a variant of canonical analysis). There is no implementation of it, as far as I know, and you would have to start from scratch (there for example). It is a linear method that finds the best linear combination of features that matches your outcome(s). This blog post shows an example of how it can be used.

Answer 4

I use Information Gain (also known as Mutual Information). My advisor and I regularly use the approach described in this paper Cohen, 2008 for analyzing features for classification by SVM.

Answer 5

Random Forests could be pretty handy for what you want to do. The randomForest package for R has a function that calculates 2 measures of importance. It also has the ability to create some partial dependence plots so you can visually inspect the marginal effect the predictor may have on the response.

Answer 6

I'm being nasty here but for a reason. Have you thought on replacing the non-uniform observations by an indicator variable present|not_present? From your description it looks like this indicator value is a valid feature since the presence of the factors D to H are non-informative: that is their presence just indicates larger outcomes.