1
$\begingroup$

I need to find a computer-driven way to come up with a model in logistic regression for an exploratory study using R. Usually I would just use the leaps or bestglm packages to find the best subset, but I have more than 300 potential predictor variables so none of them can go through the 2^300 potential models.

Are there any other computer-driven methods to select covariates for logistic regression that someone might recommend? If you could suggest the R package that implements them would be ideal.

What I have been doing so far is running bestglm in groups of 10 predictors at a time, taking note of the ones that appear in the best models and hope that I can reduce the demand on computer power. Not ideal, but I don't know how else to tackle this.

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ You could do worse than to use LASSO. $\endgroup$
    – Sycorax ♦
    Commented Feb 17, 2016 at 23:19
  • $\begingroup$ I tried using the example of glmnet found here: web.stanford.edu/~hastie/glmnet/glmnet_alpha.html but, after following the steps suggested, the model selected was the intercept-only model. Just by this rudimentary approach I'm using I have found much better models so I'm not sure if LASSO would work for me here. $\endgroup$
    – Adam
    Commented Feb 17, 2016 at 23:33
  • 2
    $\begingroup$ If you're only getting an intercept back, that's telling you that none of the variables have a strong linear relationship with the outcome. Perhaps some interactions or basis expansions could work but basically your data is not strongly informative enough about the features that you have to justify anything more than an intercept model. The lasso is a principled approach to this problem, while what you're suggesting has serious flaws. Construct a train/test partition and compare true to predicted outcome values to see what I mean. $\endgroup$
    – Sycorax ♦
    Commented Feb 18, 2016 at 0:15

2 Answers 2

Reset to default
4
$\begingroup$

This represents questionable statistical practice. On what statistical principle is that algorithm based? Why do you need to do "model selection" as opposed to stating a model and fitting it? Your approach will result in quite volatile models, i.e., you will find that bootstrapping the process would reveal a great deal of confusion about which predictors are selected.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Well, these are the very early stages of a very exploratory study. Currently we have little guidance or purpose aside from "find relationships among this variable and these +300 predictors". My objective is therefore to find an algorithm that selects the best subset of predictors that best classify predicted VS actual cases in the dependent variable. If I throw all the predictors it classifies at about 20% success rate. But I have (laboriously) selected 10 that classify at around 11% success. So that makes me think if only 10 out of 300 are doing such a great job, I may just need to... $\endgroup$
    – Adam
    Commented Feb 17, 2016 at 23:40
  • $\begingroup$ ... look a little bit more to get me closer to the 20% classification success that I would get if I used everything. $\endgroup$
    – Adam
    Commented Feb 17, 2016 at 23:40
  • 5
    $\begingroup$ I know that's what you want to do but this is a disastrous approach if you care about model validation and stability of selected predictors. Also you are using a discontinuous improper accuracy scoring rule, will will cause you to select the wrong features and give them the wrong weights. $\endgroup$
    – Frank Harrell
    Commented Feb 17, 2016 at 23:48
  • $\begingroup$ Did you try random forrest to get some idea about what predictive accuracy you can expect, and to see if interactions are important? Or lasso, with interactions included? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 5, 2017 at 20:56
2
$\begingroup$

Are you trying to examine the effects within your sample (ie. explanatory model) or are you using this data as a training set to build a model that can be used externally (ie. predictive model)?

If the goal is to "find relationships among this variable and these 300+ predictors", I tend to agree with a regularized regression approach like LASSO and trying to establish some sort of "variable importance hierarchy". It's possible that all your variables are poor predictors, but I believe you can vary a cut-point in the LASSO to establish a relative importance. Multicollinearity may be an issue here, in which case you can try something like principal components regression or structured equation modeling approach.

If you're trying to build a predictive model, as others have mentioned, you cannot train the model AND assess model performance on the same dataset by simply using % accuracy without an external validation method (eg. train/test set split, cross validation, bootstrap). Random forest seems like a good option here, given the many predictors.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Minor correction: the bootstrap and cross-validation are internal validation methods. But they can be done rigorously and often outperform external validation. $\endgroup$
    – Frank Harrell
    Commented Jan 5, 2017 at 22:06
  • $\begingroup$ @FrankHarrell good catch! $\endgroup$
    – dwhdai
    Commented Jan 5, 2017 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.