# Testing for coefficients significance in Lasso logistic regression

I have fit a logistic regression model with L1 regularization (Lasso logistic regression) and I would like to test the fitted coefficients for significance and get their p-values. I know Wald's tests (for instance) are an option to test the significance of individual coefficients in full regression without regularization, but with Lasso I think further problems arise which do not allow to apply the usual Wald formulas. For instance, the variance estimates neded for the test do not follow the usual expressions. The original Lasso paper:

Regression Shrinkage and Selection via the Lasso

suggests a bootstrap-based procedure to estimate the coefficients variance, which (again, I think) may be needed for the tests (section 2.5, last paragraph of page 272 and beginning of 273):

One approach is via the bootstrap: either $$t$$ can be fixed or we may optimize over $$t$$ for each bootstrap sample. Fixing $$t$$ is analogous to selecting the best subset (of features) and then using the least squares standard error for that subset

What I understand is: fit a Lasso regression repeatedly to the whole dataset until we find the optimal value for the regularization parameter (this is not part of the bootstrap), and then use only the features selected by the Lasso to fit OLS regressions to subsamples of the data and apply the usual formulas to compute the variances from each of those regressions. (And then what should I do with all those variances of each coefficient to get the final variance estimate of each coefficient?)

Furthermore, is it correct to use the usual significance tests (for instance Wald's test which makes use of the estimated betas and variances) with the Lasso estimates of the coefficients and the bootstrap-estimated variances? I am fairly sure it is not, but any help (use a different test, use a more straightforward approach, whaterever...) is more than welcome.

According to the answers here I suspect inference and p-values just cannot be obtained. In my case, p-values are an external requirement (although the use of L1 regularization was my choice).

Thanks a lot

EDIT What if I fit an OLS logistic regression using only the variables selected by a previous run of the Lasso logistic regression? Apparently (see here),

There's no need to run the model again after doing cross-validation (you just get the coefficients from the output of cv.glmnet), and in fact if you fit the new logistic regression model without penalisation then you're defeating the purpose of using lasso

But what if I do this with the sole purpose of being able to compute p-values while keeping the number of variables low? Is it a very dirty approach? :-)

• To do inference for LASSO models you can also check out the CRAN package hdi which provides inference for high-dimensional models, you might want to take a look at that... Commented Apr 6, 2017 at 15:16
• The full methods are well described in this paper: projecteuclid.org/euclid.ss/1449670857 Commented Apr 6, 2017 at 15:41
• And there's also package cran.r-project.org/web/packages/selectiveInference/index.html that may be useful to provide inference for the LASSO... Commented Mar 6, 2018 at 21:57

The issue with performing inference after model selection is that you are selecting the most predictive variables and then performing inference as if they were selected independently of the data. It is possible to show that refitting the regression model after doing model selection with the lasso (or any other model selection method!) may lead to $$\sqrt{n}$$-biased estimates (which is one reason why a simple Gaussian approximation will often fail for confidence intervals)