Decision Tree as variable selection for Logistic Regression

Question

I have to do a Logistic Regression, and have to use a subset of the variables. I received this "tip": do a Decision Tree first, and use the most relevant variables in the Logistic Regression.

Is this a valid technique? If not, what problems does it face?

PS: This tip is also given here for a standard regression.

UPDATE: First of all thanks for all the answers. I'm definitely going to try the LASSO model. But, the algorithm must be simple, and easy for the IT to implement. It's a "millions of lines" kind of database, and the model needs to recurrently be estimated. Does it change the approach?

Answer 1

If you have access to LASSO and your predictors are all numeric then that is a good choice as Peter mentioned.

If you have a massive number of predictors as experienced often in fields like marketing - then this can be computationally too expensive. In that case, a tree can be used, but a random forest or gradient boosted regression tree would likely be better choices as variable importance is more robust (for the same reason boosted and bagged trees are expected to be more stable).

The party package in R might be another good choice, as the conditional inference trees and associated forests and variable importance measures are purported to be less biased. Party VI

Outside of statistics, Googling "Feature Selection" might give you more ideas.

Answer 2

In addition to the usual problems with automatically choosing variables, this uses trees for a purpose they were not designed for. To me, there are two big advantages to classification trees:

1) They are intuitively very clear. 2) They allow you to look at interactions in ways that would be very difficult in a regression model, because the interactions can be different in different branches of the tree.

Variable selection is a big topic and has been often discussed here. I am partial to LASSO and or LAR, myself.

Answer 3

The random forest technique is related to decision trees. A metric that it outputs is a variable importance measure. This measure of often used for feature selection, which is a technique to select a subset of variables. They key aspect to understand is that there are many ways to go wrong with feature selection (subset selection). For example, if your model is assessed with a resampling plan (cross validation/bootstrap), you must repeat the variable selection at each iteration. This requires a good amount of background reading to fully appreciate. But searching this site and others for "randomForest", "variable importance", "variable selection", "cross-validation", and "overfitting" will get you started.

Answer 4

One of the problems with this approach is that logistic regression and decision trees are very different algorithms, so the set of features that work well with a decision tree are not necessarily going to be the features that work well with a logistic regression model (and vice versa). So my advice would be that this approach is a bit of a hack and there are likely to be better approaches. There is a good tutorial on feature selection:

Isabelle Guyon, André Elisseeff, "An Introduction to Variable and Feature Selection", Journal of Machine Learning Research, 3(Mar):1157-1182, 2003. (www)

I, like Peter Flom (+1) and B_Miner (+1), am quite keen on LASSO/LARS based methods (The Random Forest is also a good algorithm), my own contribution to this can be found here:

Gavin C. Cawley and Nicola L. C. Talbot, Gene selection in cancer classification using sparse logistic regression with Bayesian regularization, Bioinformatics, (2006) 22 (19): 2348-2355. (www)

where the regularisation parameter for a LASSO type penalty is integrated out analytically, so there are no hyper-parameters to tune.

As @julieth points out (+1) if you use feature selection, you must perform the feature selection step independently in each fold of the cross-validation procedure, or you will end up with an optimistically biased performance estimate. See this paper for details

Christophe Ambroise and Geoffrey J. McLachlan, Selection bias in gene extraction on the basis of microarray gene-expression data, PNAS, vol. 99, no. 10, pp 6562–6566, 2002 (www)

This paper is a "must read" for anybody that is interested in feature selection!