What is the correct way to report the results of a penalized (Ridge, LASSO, ElasticNet) logistic regression model?

Question

I was wondering what would be a sensible way to report the results of a penalized logistic regression model in a scientific article (in terms of coefficients, metrics, diagnostics, hyperparameters, etc.)?
With an explanatory (unpenalized) model, I would report things like estimated coefficients, standard errors, degrees of freedom, (pseudo-)R2, p-values, test statistics, perhaps some 2D/3D plot... Here, I'm at a loss.

Some context: I'm working with an ElasticNet model (with glmnet and caret in R) but I guess answers apply equally to Ridge or LASSO models. My initial interest was to do inference. Yet, for various reasons (detailed here) I cannot use a "regular" logistic model to identify what explanatory variables drive my outcome so instead, I've decided to build a penalized logistic model in order to find values for my predictors beyond which the probabilities of my outcome cross some threshold values (and/or interpret coefficients directly) and make some quantified recommendations to obtain the desired outcome.

Answer 1

The best way to proceed--both from your and your readers' perspectives--is to repeat your entire modeling process on multiple bootstrap samples of your dataset (re-samples with replacement of the same size as your dataset) and report a summary of those results. That's explained in the context of LASSO for linear regression in Section 6.2 of Statistical Learning with Sparsity, but directly extends to elastic net for logistic regression if you use an appropriate measure for cross-validated selection of $\alpha$ (the ridge versus LASSO tradeoff) and $\lambda$ (penalization), like deviance (log loss). Don't rely on improper scoring rules like accuracy for building your model.

The idea of the bootstrap principle is that resampling with replacement from your dataset mimics the process of taking your dataset from the underlying population. So, for example, the (co)variances of coefficient values among models based on multiple bootstrap samples provides an estimate of the (co)variances of coefficient estimates from the model you built on the full data set. Similar arguments apply to any measure of modeling performance.

The third and fourth paragraphs of this answer outline the approach in the context of a question on elastic net. This can directly provide further evaluation of modeling bias via the optimism bootstrap.

With any predictor-selection method, you will typically find that the retained predictors will differ among bootstrapped samples. That's OK, as you would find the same behavior if you took multiple full samples from the population. Use that result as a caution against the idea that LASSO or elastic net always return the "most important" predictors.

Finally, remember that the above estimates the performance of your modeling process, not necessarily that of your model. It provides no protection, for example, if you have a limited data sample that doesn't reliably represent the underlying population of interest.