Report coefficients in elastic net regression after cross validation

Question

I believe several variants of this question have been discussed previously, but I'm still not clear on the appropriate approach so I wanted to ask my specific question here.

I've used elastic net regression to train a classifier (logistic regression) on a set of ~54000 observations, each with ~200 features. I have significant class imbalance (10:1 negative:positive observations). I've used a nested cross-validation strategy (10-fold inner loop to select elastic net hyperparameters, 10-fold outer loop to assess prediction error), and I'm now confused as to 1) how to summarize the model coefficients across cross validation folds (i.e., a valid way to report a single estimate for each coefficient), 2) if and how I can report any confidence statements about the final coefficients.

Basically, the classifier works quite well, and I'd like to be able to interrogate the coefficients to understand why (which variables are strong predictors, and what their relative effects are).

My intuition would be to just report the 10 values for each coefficient obtained from my final (outer) cross validation, selecting the median (or mean) as my best estimate. But this seems like a similar strategy to generating bootstrap estimates for my model coefficients (e.g., after selecting hyperparameters, fit the model to 100 resamples of the full dataset), and my reading of additional posts indicates does not give valid confidence intervals.

Thanks for any input!

Answer 1

Regarding your question about post-selection confidence intervals for the LASSO look for a paper by Tibshirani et al. titled "Exact Post-Selection Inference for Sequential Regression Procedures". As for implementation, look for a R package called selectiveInference (works with glmnet). There is also a Python implementation, you can find information on both at the github page https://github.com/selective-inference/ Hope it helps.

Answer 2

I am facing a similar issue as the OP, reporting coefficients of elastic net. In a comment, he wrote that it is wrong to report coefficients made in Cross Validation, and that one should train a model on all the data and report those.

I would argue the opposite: Do several CV folds, and report the average coefficient and the variance. Do this because these are the models with which you achieved your error. If you fit the model on all the data, elastic net can select a certain set of features which might not match what was selected in CV folds. So you can end up reporting an error and coefficients that don't actually match. This is only okay if you see that elastic net selects the same features in every CV fold and the variance is low.