# Feature construction in R

I am wondering if there are any algorithms (perhaps genetic algorithms) in R for feature construction (deriving candidate predictors from existing predictors)? I am thinking of a routine to test higher order powers, interactions, ratios, and linear combinations and nonlinear functions of existing variables (sin, cos, atan etc).

This could be a filter or wrapper routine (i.e. not using a learning algorithm or using one to define fitness of the feature).

My goal is to "discover" potentially meaningful ratios and the like of existing predictors.

Thanks!

It would seem to me that this would leave you highly vulnerable to problems like spurious correlation and even overfitting. I forget the name of the principle that states the more models you try, the greater your risk of stumbling upon a bad one -- if you try so many models as to actually run a genetic algorithm, you can imagine how that principle is violated.

• I think the OP is talking about using a genetic algorithm to extract or engineer features, not to find models. – Aman Jan 1 '14 at 19:56

You could go about it like this: starting from a data.frame, you add a 'reasonable' set of transformed predictors or even interactions to your data (model.matrix and similar should be able to pull this off).

Once you're there, any variable selection method could do. glmnet comes to mind, but there are many options. A disadvantage to this way of working is that it will be hard to ensure that the main effect is in the model when an interaction is. Perhaps some forms of variable selection support this, but I know of no obvious ones besides stepwise procedures (that would defy the purpose).

You could start with something simple like finding principle components or independent components. You could also get a little crazy and generate all the 2-way interactions of your variables. Obviously, as you generate and test more features, you need a feature selection algorithm that is more robust against overfitting.

Some modeling algorithms, like MARS, random forests, and non-linear SVMs automatically find certain interactions among your original features.

• (+1) It seems to me that the problem with RFs or SVMs (and more generally, any ensemble methods) is that such interactions are rarely interpretable in a direct manner. Some variations around bagging that allow to perform feature selection on specific type of predictors have been proposed, see e.g. the logicFS package. – chl Aug 19 '11 at 18:49