I believe several variants of this question have been discussed previously, but I'm still not clear on the appropriate approach so I wanted to ask my specific question here.

I've used elastic net regression to train a classifier (logistic regression) on a set of ~54000 observations, each with ~200 features. I have significant class imbalance (10:1 negative:positive observations). I've used a nested cross-validation strategy (10-fold inner loop to select elastic net hyperparameters, 10-fold outer loop to assess prediction error), and I'm now confused as to 1) how to summarize the model coefficients across cross validation folds (i.e., a valid way to report a single estimate for each coefficient), 2) if and how I can report any confidence statements about the final coefficients.

Basically, the classifier works quite well, and I'd like to be able to interrogate the coefficients to understand why (which variables are strong predictors, and what their relative effects are).

My intuition would be to just report the 10 values for each coefficient obtained from my final (outer) cross validation, selecting the median (or mean) as my best estimate. But this seems like a similar strategy to generating bootstrap estimates for my model coefficients (e.g., after selecting hyperparameters, fit the model to 100 resamples of the full dataset), and my reading of additional posts indicates does not give valid confidence intervals.

Thanks for any input!

  • $\begingroup$ I think I've answered my own question from looking at additional posts: 1) I'm confusing cross-validation for model selection with bootstrapping for parameter estimation. There's no summarizing of coefficients over cross-validation folds. Rather, you use cross-validation to assess the error of a specific model form, but the coefficients in the final model are estimated using all the data. 2) Estimating confidence intervals for lasso coefficients is an open problem (still not sure if this is correct). 3) Naively bootstrapping lasso coefficients does not give valid confidence estimates. $\endgroup$ – pseudo Jun 16 '17 at 11:18

Regarding your question about post-selection confidence intervals for the LASSO look for a paper by Tibshirani et al. titled "Exact Post-Selection Inference for Sequential Regression Procedures". As for implementation, look for a R package called selectiveInference (works with glmnet). There is also a Python implementation, you can find information on both at the github page https://github.com/selective-inference/ Hope it helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.