Selecting the best subset of variables for parsimonious binary logistic regression models

In addition to PROC VARCLUS, randomForest, glmnet, and assessing multicollinearity among potential predictor variables (without regards to the outcome of interest), I am seeking other methods of variable selection in lieu of using stepwise methods for building more parsimonious binary logistic regression models (containing 8 to 12 variables to predict outcomes such as loan payment/default or current/late payment history) from a wide array of potential predictor variables (500+ variables, 200k+ records).

Below I have included an R script using FSelector to select the 8 highest "ranked" variables:

library(FSelector)
fit <- information.gain(outcome ~ ., dataset)
fit2 <- cutoff.k(fit,8)
reducedmodel <- as.simple.formula(fit2,"outcome")
print(reducedmodel)


I have two questions regarding this script and the FSelector algorithm in general:

1. Is the information.gain criteria in the above script synonymous with Kullback-Leibler divergence? If so, can someone explain this in more layman terms than Wikipedia as I am relatively new to this area of statistics and would like to start off with the right idea of this concept as I may likely use this approach a great deal in the future?

2. Is this a valid approach, if there is such a thing as a valid approach, to select a desired number of variables for a binary logistic regression model (e.g., selecting the 8 highest "ranked" variables for use in a parsimonious model)? If not, can you provide an alternative approach to do so?

Any insight or references regarding this topic and/or these questions will be greatly appreciated!

Variable selection without penalization is invalid.

• If you don't want to do variable selection (which will hurt predictive discrimination) but just want good predictions, there are many packages available. For example ridge (L2 norm) penalized logistic or ols regression can be done using the R rms package. – Frank Harrell Oct 3 '13 at 19:22
• It would be interesting to know why you need variable selection. And when doing so you may have an additional obligation to show that the variables selected are not arbitrary. This can be shown by bootstrapping the entire process and checking for consistency of lists of variables. You may be disappointed to find that the list is not as stable as you would have guessed. – Frank Harrell Oct 4 '13 at 11:16
• With computers I don't see such predictive accuracy-wasting parsimony as being very necessary. Unless of course fewer variables can be collected from customers in the first place. – Frank Harrell Oct 4 '13 at 16:57
• Depending on your effective sample size, penalization may be needed. The need for penalization is somewhat independent of whether you do variable selection. It would be worthwhile using the optimism bootstrap or a huge independent validation sample to estimate the loss in $R^2$ from doing variable selection over fitting a full model. – Frank Harrell Oct 4 '13 at 18:46
• If you estimate maximum number $p$ of parameters fitted or examined for fitting, one rule of thumb is that if $p < 10000/15$ you are unlikely to need to penalize unless perhaps you are doing variable selection. $p$ needs to count terms examined but dropped, nonlinear terms, all dummy variables, and all interaction terms. – Frank Harrell Oct 4 '13 at 19:07

Rarely does a question appear that I believe I can answer.

This isn't in R (at least I don't believe there is a addin for it). It doesn't answer your exact questions either, although I believe may be of help.

I am seeking other methods of variable selection in lieu of using stepwise methods for building more parsimonious binary logistic regression models (containing 8 to 12 variables to predict outcomes such as loan payment/default or current/late payment history) from a wide array of potential predictor variables (500+ variables, 200k+ records).

Predictor Selection Using the CCR/Step-Down Algorithm In step 1 of the step-down option, a model containing all predictors is estimated with K* components (where K* is specified by the user or determined by the program if the Automatic option is activated), and the relevant CV statistics are computed. In step 2, the model is then re-estimated after excluding the predictor whose standardized coefficient is smallest in absolute value, and CV statistics are computed again. Note that both steps 1 and 2 are performed within each subsample formed by eliminating one of the folds. This process continues until the user-specified minimum number of predictors remains in the model (by default, Pmin = 1). The number of predictors included in the reported model, P*, is the one with the best CV statistic.

Here is the summarization for when it was presented at the Paris workshop on PLS developments 2 years ago.

Jay Magidson. "Correlated Component Regression A Sparse Alternative to PLS Regression." http://statisticalinnovations.com/technicalsupport/ParisWorkshop.pdf

It sounds like what you are partially trying to accomplish is done with this algorithm in this tutorial:

"Using Correlated Component Regression with a Dichotomous Y and Many Correlated Predictors." http://statisticalinnovations.com/products/xlstattutorials/XLCCRtutorial2.pdf

I also hate it when people answer questions by just referencing other works instead of actually answering the question, but, I just feel this is the best answer if I'm understanding everything correctly. Hope it helps.

• I appreciate your recommendation of correlated component regression; however, XLSTAT appears to be the only program that caters to this type of analysis. In relation to other statistical programs, this program is very very inexpensive, which is great (not all programs can be a cheap as R!). Nonetheless, I am restricted to using SAS and R as well as an in-house program to build my models so this type of analysis is unfortunately not very helpful for me. I hope this thorough post helps others as I may look into XLSTAT in the future. Thanks Taal! – Matt Reichenbach Oct 3 '13 at 14:20
• @MattReichenbach No problem, although they do offer a 90 day free trial of pretty much the full version of their software. The only limitation that I found is when doing CCR, you can't analyze more than 20 columns at a time. Regardless though, if you have some data and you can put it in speadsheet form for me, I'll run a CCR on it and give you the results back. Just open a chat with me. And no, I don't work for XLSTAT or have any affiliation with them at all as it probably seems from my posts. – Taal Oct 3 '13 at 16:35
• they should be payinig you though as you are selling their product well! Haha, thank you for the offer; however, 20 columns is far too few for the data that I typically work with (typically 500-1500 columns), and I don't want to waste your time as it would require 25+ runs to assess all of the variables for a given model – Matt Reichenbach Oct 3 '13 at 16:41
• @MattReichenbach To be honest it would help me to help you ( it would help solidify some concepts I'm learning atm), plus you may be able to answer a question or two for me. Regardless, open a chat, I even have easy access to a supercomputer we can run it on together. Only reason I'd see otherwise is if your data is proprietary in some way. – Taal Oct 3 '13 at 16:57
• @MattReichebach lol now im sounding pushy, but I also deal with PII (personable identifiable information) but all you have to do is recode that to an obsv id – Taal Oct 3 '13 at 17:16

To answer the above two questions:

1) In layman terms, Kullback-Leibler divergence, as displayed in R when using the FSelector package, is the relative amount of information that can be gained by using a given potential predictor variable.

2) It is NOT a valid approach to select a desired number of variables for a binary logistic regression model (e.g., selecting the 8 highest "ranked" variables for use in a parsimonious model) as no such approach exists. Most importantly, it does not take into account collinearity among predictor variables, as it simply rank orders variables by information gain. Information gain is very similar to the c-statistic or information value in terms of being a univariate measurement for classification strength for a given predictor variable. As such, it is useful to rank variables, but other methods must also be applied to further build parsimonious models (eg. PCA and Lasso or Ridge Regression).

I hope this helps someone!