3
$\begingroup$

For those of you familiar with Exploratory Factor Analysis (EFA) and Random Forest (RF), I have recently had an idea of combining these two methods to reduce the number of potential predictor variables for use in a parsimonious binary logistic regression model. For the purposes of this post, assume large n (200k or more) and 1000 potential predictor variables.

To employ this idea, the first step would be to perform an EFA with all potential predictor variables using proc varclus. Additionally, using randomForest to rank all potential predictor variables by IncNodePurity (Gini Index).

After these two methods are independently used, I propose retaining the variable with the largest IncNodePurity (from RF) within each factor (from EFA).

Does anyone have any thoughts/concerns with this methodology (or lack thereof) for feature selection? I am aware that this "picking and choosing" of methods may be complete garbage, but I had this random thought and wanted to share. Thanks!

Improve this question
$\endgroup$
  • $\begingroup$ @Momo, good question, for an example, I have a dataset with 140 variables, the range of IncNodePurity falls between 0.03 and 96.36. Three of the variables at the top of the IncNodePurity rankings are highly correlated (would likely load into the same factor if I performed an EFA). This is why I posted this, to avoid asking another question, but if you have an answer to how I could deal with this situation, I would appreciate your post. $\endgroup$ – Matt Reichenbach Sep 3 '13 at 18:14
  • $\begingroup$ My goal is to reduce the number of potential predictor variables to fewer than 15 variables, which have to be used in a logistic regression model (thus correlation is a major concern with using random forest results alone), but I agree with your thoughts on how random forests handle these high-dim cases extraordinarily well! $\endgroup$ – Matt Reichenbach Sep 3 '13 at 18:24
  • 2
    $\begingroup$ Hm, I see. Why not try logistic regression with elastic net penalty than? You can cut the regularization path at the regularization parameters where you have exactly 15 predictors. The $L_2$ regularization takes care of the correlations and the $L_1$ regularization will select the fifteen predictors. The reason I'm skeptical with pipelines is that often it is better to solve the problem directly. 200k and 1000 predictors should be feasible computationally. $\endgroup$ – Momo Sep 3 '13 at 18:30
  • 1
    $\begingroup$ The advantage of this idea is that you would optimize the logistic regression loss function directly subject to the predictor selection. You can search this site for the tags elastic-net and lasso for a starter. In R there is the glmnet package that implements this (check this presentation moseslab.csb.utoronto.ca/alan/glmnet_presentation.pdf), and see also the paper here jstatsoft.org/v33/i01/paper (page 8 ff is exactly your problem). I don't know anything about SAS though. $\endgroup$ – Momo Sep 3 '13 at 18:46
  • 1
    $\begingroup$ Thanks @Momo, I will review the presentation and paper and will likely come up with another question. Although I am interested to see if this current question gets any other feedback, but I really appreciate your help! +1 $\endgroup$ – Matt Reichenbach Sep 3 '13 at 19:01
1
$\begingroup$

Thanks to @Momo, I have came to the conclusion that making use of L2 regularization to handle collinearity that likely exists among potential predictor variables and L1 regularization to allow me to select the desired number of variables to create my logistic regression models is a better approach than combining EFA and Random Forest. Thanks @Momo for the nudge over to elastic net and the documents you provided to start me on the right track!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.