1
$\begingroup$

I believe several variants of this question have been discussed previously, but I'm still not clear on the appropriate approach so I wanted to ask my specific question here.

I've used elastic net regression to train a classifier (logistic regression) on a set of ~54000 observations, each with ~200 features. I have significant class imbalance (10:1 negative:positive observations). I've used a nested cross-validation strategy (10-fold inner loop to select elastic net hyperparameters, 10-fold outer loop to assess prediction error), and I'm now confused as to 1) how to summarize the model coefficients across cross validation folds (i.e., a valid way to report a single estimate for each coefficient), 2) if and how I can report any confidence statements about the final coefficients.

Basically, the classifier works quite well, and I'd like to be able to interrogate the coefficients to understand why (which variables are strong predictors, and what their relative effects are).

My intuition would be to just report the 10 values for each coefficient obtained from my final (outer) cross validation, selecting the median (or mean) as my best estimate. But this seems like a similar strategy to generating bootstrap estimates for my model coefficients (e.g., after selecting hyperparameters, fit the model to 100 resamples of the full dataset), and my reading of additional posts indicates does not give valid confidence intervals.

Thanks for any input!

$\endgroup$
  • $\begingroup$ I think I've answered my own question from looking at additional posts: 1) I'm confusing cross-validation for model selection with bootstrapping for parameter estimation. There's no summarizing of coefficients over cross-validation folds. Rather, you use cross-validation to assess the error of a specific model form, but the coefficients in the final model are estimated using all the data. 2) Estimating confidence intervals for lasso coefficients is an open problem (still not sure if this is correct). 3) Naively bootstrapping lasso coefficients does not give valid confidence estimates. $\endgroup$ – pseudo Jun 16 '17 at 11:18
3
$\begingroup$

Regarding your question about post-selection confidence intervals for the LASSO look for a paper by Tibshirani et al. titled "Exact Post-Selection Inference for Sequential Regression Procedures". As for implementation, look for a R package called selectiveInference (works with glmnet). There is also a Python implementation, you can find information on both at the github page https://github.com/selective-inference/ Hope it helps.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I am facing a similar issue as the OP, reporting coefficients of elastic net. In a comment, he wrote that it is wrong to report coefficients made in Cross Validation, and that one should train a model on all the data and report those.

I would argue the opposite: Do several CV folds, and report the average coefficient and the variance. Do this because these are the models with which you achieved your error. If you fit the model on all the data, elastic net can select a certain set of features which might not match what was selected in CV folds. So you can end up reporting an error and coefficients that don't actually match. This is only okay if you see that elastic net selects the same features in every CV fold and the variance is low.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.