covariate selection in inference problems in logistic regression

Question

For my specific problem, but a common situation in the medical field, I have several hundred patients, and about 10-20 exaplnatory variables. the goal is to examine a specific predictor("treatment") for mortality.

considering a 20% event rate you'd get a high EPV, so it's desired to have a simpler model/less features to avoid overfitting or noise variables. It's rather pervasive to use univariate selection + stepwise for inference ("After adjustment, treatment A was significantly associated with reduced mortality"). From reading in CV, and also from Steyerberg/Harrell/ISLR and other online sources, this is generally a mistake, perhaps unless you have a very large sample size. The options I generally see, other than "stepwising":

1. Using penalization/shrinkage (e.g. lasso) allows variable selection, but this does not readily translate for inference (in contrast to just getting coefs for prediction)
2. Using propensity score for lumping covariates together (unless a risk score is available) - such as baseline characteristics.
3. Using some sort of dimension reduction like PCA which I'm not familiar with

I can't select background characteristics based solely on literature (all might be relevant). Am I correct in my assessment? how do I make the best inference regarding the "treatment" variable?

Answer 1

Some methods have been developed for testing the significance of LASSO weights. For example:

[1] Lockhart et al. (2014). A significance test for the LASSO.

[2] Meinshausen et al. (2009). p-Values for high-dimensional regression.

[3] Brink-Jensen and Ekstrom (2014). Inference for feature selection using the Lasso with high-dimensional data.

These papers consider linear regression, but the procedures in [2] and [3] are pretty general, and might also apply to logistic regression (not sure about [1]). The papers include some citations for related methods.

The PCA approach might work, but there are some caveats. PCA is an unsupervised method that tries to find a good representation of the data with reduced dimensionality. It doesn't have any regard for the how well the reduced representation predicts external variables. Projecting your data into the subspace selected by PCA might give good predictions. But, there are no guarantees, and PCA can throw out relevant information if it happens to lie along low-variance directions. Another caveat is that you'll have to select the number of reduced dimensions. If you do this using information about the variable to be predicted, then you can't use standard techniques to test the significance of your regression weights. The reason is that these techniques aren't aware of the earlier model selection step. It may still be possible to perform inference, but you'd be in the same boat (for the same reasons) as LASSO.

This paper has some nice discussion about testing the significance of estimated parameters when a model selection procedure has been used on the same data:

Chatfield et al. (1995). Model Uncertainty, Data Mining and Statistical Inference.