# Variable selection for logistic regression with separated data

I have a fairly large dataset ($\approx 3 \bar{M}$ observations for a dozen candidate predictors) and I would like to perform a logistic regression on that dataset. I have a problem of separation in that dataset so usual model can't converge. That's why I am using Firth penalization (logistf package for R) to have my model to adjust.

I would like to select the best subset of variables for my final model but I can't find the proper way to do that. I know that stepwise selection is out of question and I usually would use L1 or L2 penalized regression so that some coefficients are reduced to 0.

My problem is : the function I am using to adjust my model doesn't handle extra penalization so no Elasticnet-Firth regression.

Is there, apart from stepwise selection, another way to select my variables?

• Posts bearing the Hauck-Donner Effect tag may also be of interest. – Sycorax says Reinstate Monica Sep 13 '16 at 23:59

In my experience with logistic regression where $n >> p$, using $L_1$ or $L_2$ regularization on the coefficients has little to no effect on inference. Variables that are selected by $L_1$ will usually have enormous $z$-scores - suggesting they remain in any final model. Furthermore, post-selection inference, or interpretation of $p$-values after performing variable selection, is a difficult and active research field. Since you seem to have ample data, regularization only complicates things.
We can select the most important features using H2O. With H2o we can use all of your data without sampling. one option is using GBM algorithm automatically calculates variable importance and output the absolute and relative predictive strength of each feature in the prediction task.