1
$\begingroup$

I was wondering what would be a sensible way to report the results of a penalized logistic regression model in a scientific article (in terms of coefficients, metrics, diagnostics, hyperparameters, etc.)?
With an explanatory (unpenalized) model, I would report things like estimated coefficients, standard errors, degrees of freedom, (pseudo-)R2, p-values, test statistics, perhaps some 2D/3D plot... Here, I'm at a loss.

Some context: I'm working with an ElasticNet model (with glmnet and caret in R) but I guess answers apply equally to Ridge or LASSO models. My initial interest was to do inference. Yet, for various reasons (detailed here) I cannot use a "regular" logistic model to identify what explanatory variables drive my outcome so instead, I've decided to build a penalized logistic model in order to find values for my predictors beyond which the probabilities of my outcome cross some threshold values (and/or interpret coefficients directly) and make some quantified recommendations to obtain the desired outcome.

$\endgroup$
6
  • 1
    $\begingroup$ What results do you mean? What do you want to present, and to whom will you present them? $\endgroup$ – Dave May 14 at 17:54
  • $\begingroup$ What kind of information are you trying to learn by using the model? $\endgroup$ – Sycorax May 14 at 18:01
  • $\begingroup$ You're both right, it was not very clear (I'm tired). I've edited my question to bring more context and precisions. I've also edited the linked question for it to be more focused in the hope someone will re-open it. Sorry for the inconvenience and thank you for your help. $\endgroup$ – Fanfoué May 14 at 18:56
  • 1
    $\begingroup$ You say "My interest lies in inference." Yet when you say you want to "make some quantified recommendations to obtain the desired outcome," it sounds more like prediction. Inference is not easy with elastic net logistic regression; presentation of a predictive model can be much more straightforward. $\endgroup$ – EdM May 14 at 22:07
  • $\begingroup$ @EdM, I initially wanted to do inference but, unfortunately, it was impossible. I changed my sentence a little bit to improve clarity on this point. I hear you when you say that presenting a predictive model can be much more straightforward, but how? What would you typically report? Only estimated coefficients with no measures of variability/uncertainty? $\endgroup$ – Fanfoué May 15 at 11:29
3
$\begingroup$

The best way to proceed--both from your and your readers' perspectives--is to repeat your entire modeling process on multiple bootstrap samples of your dataset (re-samples with replacement of the same size as your dataset) and report a summary of those results. That's explained in the context of LASSO for linear regression in Section 6.2 of Statistical Learning with Sparsity, but directly extends to elastic net for logistic regression if you use an appropriate measure for cross-validated selection of $\alpha$ (the ridge versus LASSO tradeoff) and $\lambda$ (penalization), like deviance (log loss). Don't rely on improper scoring rules like accuracy for building your model.

The idea of the bootstrap principle is that resampling with replacement from your dataset mimics the process of taking your dataset from the underlying population. So, for example, the (co)variances of coefficient values among models based on multiple bootstrap samples provides an estimate of the (co)variances of coefficient estimates from the model you built on the full data set. Similar arguments apply to any measure of modeling performance.

The third and fourth paragraphs of this answer outline the approach in the context of a question on elastic net. This can directly provide further evaluation of modeling bias via the optimism bootstrap.

With any predictor-selection method, you will typically find that the retained predictors will differ among bootstrapped samples. That's OK, as you would find the same behavior if you took multiple full samples from the population. Use that result as a caution against the idea that LASSO or elastic net always return the "most important" predictors.

Finally, remember that the above estimates the performance of your modeling process, not necessarily that of your model. It provides no protection, for example, if you have a limited data sample that doesn't reliably represent the underlying population of interest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.