3
$\begingroup$

After reviewing related questions on Cross Validated and countless articles and discussions regarding the inappropriate use of stepwise regression for variable selection, I am still unable to find the answers that I am looking for in regards to building parsimonious, binary logistic regression models from datasets with 1000 (or more) potential predictor variables.

For some background information, I typically work with large datasets, 500k or more rows, and my interest is in building binary logistic regression models to predict whether an individual will pay (1) or not pay (0) their bill on a particular account without using stepwise logistic regression. Currently, stepwise logistic regression is hailed as the “perfect method” among other statisticians that I have worked with, and I would like to change that as I have witnessed many of its pitfalls firsthand.

I have recently dabbled in PCA (proc varclus) and random forest analyses (randomForest) with the latter being especially helpful; however, I am still seeking further direction on how to reduce the number of variables in my binary logistic models without using stepwise logistic regression. With that being said, any help (suggested articles or thoughts) is greatly appreciated. Thanks!

$\endgroup$
  • 1
    $\begingroup$ Have you tried L1 regularized regression with transformations of any continuous variables? $\endgroup$ – alex Aug 20 '13 at 15:08
2
$\begingroup$

Tree-based methods (e.g., CART) are widely used for solving this problem. While I would always prefer to estimate a logit model using theory, when in situations where there is insufficient theory or the data is not well understood, a tree-based method is, in my experience, always preferable to a logit model as it scales better to large data sets, is much more robust and better deals with non-linearities and interactions. If you superiors are desperate for a logit model, you can use the variables selected by the tree-based model in a logit model.

$\endgroup$
  • 1
    $\begingroup$ No, tree methods that utilize a single tree often require 100,000 subjects to yield stable trees whose predictions validate well. It is the imposition of structure (e.g., additivity) that make regression models have better validated $R^2$ than single trees. If you bootstrap CART fits you'll see what I mean. $\endgroup$ – Frank Harrell Aug 20 '13 at 15:57
  • $\begingroup$ @FrankHarrell, I often work with datasets that have well over 100k subjects (with more than adequate sample size in the smallest category), so this shouldn't be a problem. Do you have any recommendations for variable selection in regards to binary logistic regression with large n and a large number of potential predictor variables? Thanks! $\endgroup$ – Matt Reichenbach Aug 20 '13 at 17:00
  • 1
    $\begingroup$ Hi @FrankHarrell I've seen you make this remark on trees before. Have you got any links or cites where I could get more info on this? The classic book on trees uses, as I recall, quite small data sets (dozens of observations, not 100's of thousands). Thanks! $\endgroup$ – Peter Flom - Reinstate Monica Aug 20 '13 at 19:08
  • $\begingroup$ The authors did not attempt stringent validations of what they produced. Some useful simulations (with code) are here: biostat.mc.vanderbilt.edu/wiki/Main/… . See also RJ Marshall, J Clin Epi 54:603; 2001 and PC Austin, Stat in Med 26:2937, 2007. $\endgroup$ – Frank Harrell Aug 20 '13 at 20:32
3
$\begingroup$

Another option is the lasso, which can efficiently select a subset of important predictors from large collection. This can be implemented in the glmnet package in R. See http://www-stat.stanford.edu/~tibs/lasso.html for more information.

$\endgroup$
1
$\begingroup$

A couple ideas: Rather than PCA, you might look into partial least squares, which tries to account for relationships both among the IVs and between the IV and DV. I've mostly used it for p >> n problems, but it could be helpful in your situation, too.

With so many variables, surely many of them are highly correlated? Since we are clearly dealing with people - I don't think you could come up with that many IVs and not have high correlations. You could check for all correlations over, say, .9, and randomly delete one of each such pair. Admittedly, an ad hoc procedure, but I've found it useful.

$\endgroup$
  • $\begingroup$ thanks Peter! I do assess these IVs for multicollinearity (without respect to the outcome of interest) as one of my initial steps. I like your idea of random deletion though! Thanks again, and I will look into partial least squares as well! $\endgroup$ – Matt Reichenbach Aug 20 '13 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.