9
$\begingroup$

[A similar question was asked here with no answers]

I have fit a logistic regression model with L1 regularization (Lasso logistic regression) and I would like to test the fitted coefficients for significance and get their p-values. I know Wald's tests (for instance) are an option to test the significance of individual coefficients in full regression without regularization, but with Lasso I think further problems arise which do not allow to apply the usual Wald formulas. For instance, the variance estimates neded for the test do not follow the usual expressions. The original Lasso paper

http://statweb.stanford.edu/~tibs/lasso/lasso.pdf

suggests a bootstrap-based procedure to estimate the coefficients variance, which (again, I think) may be needed for the tests (section 2.5, last paragraph of page 272 and beginning of 273):

One approach is via the bootstrap: either $t$ can be fixed or we may optimize over $t$ for each bootstrap sample. Fixing $t$ is analogous to selecting the best subset (of features) and then using the least squares standard error for that subset

What I understand is: fit a Lasso regression repeatedly to the whole dataset until we find the optimal value for the regularization parameter (this is not part of the bootstrap), and then use only the features selected by the Lasso to fit OLS regressions to subsamples of the data and apply the usual formulas to compute the variances from each of those regressions. (And then what should I do with all those variances of each coefficient to get the final variance estimate of each coefficient?)

Furthermore, is it correct to use the usual significance tests (for instance Wald's test which makes use of the estimated betas and variances) with the Lasso estimates of the coefficients and the bootstrap-estimated variances? I am fairly sure it is not, but any help (use a different test, use a more straightforward approach, whaterever...) is more than welcome.

According to the answers here I suspect inference and p-values just cannot be obtained. In my case, p-values are an external requirement (although the use of L1 regularization was my choice).

Thanks a lot

EDIT What if I fit an OLS logistic regression using only the variables selected by a previous run of the Lasso logistic regression? Apparently (see here),

There's no need to run the model again after doing cross-validation (you just get the coefficients from the output of cv.glmnet), and in fact if you fit the new logistic regression model without penalisation then you're defeating the purpose of using lasso

But what if I do this with the sole purpose of being able to compute p-values while keeping the number of variables low? Is it a very dirty approach? :-)

$\endgroup$
4
$\begingroup$

The problem with using the usual significance tests, is that they assume the null that is that there are random variables, with no relationship with the outcome variables. However what you have with lasso, is bunch of random variables, from which you select the best ones with the lasso, also the betas are shrunk. So you cannot use it, the results will be biased.

As far as I know, the bootstrap is not used to get the variance estimation, but to get the probabilities of a variable being selected. And those are your p-values. Check the Hasie's free book, Statistical Learning with Sparsity, chapter 6 is talking about the same thing. http://web.stanford.edu/~hastie/StatLearnSparsity/

Also check this paper for some other ways to get p-values from lasso https://arxiv.org/pdf/1408.4026.pdf There are probably more

$\endgroup$
3
$\begingroup$

The issue with performing inference after model selection is that you are selecting the most predictive variables and then performing inference as if they were selected independently of the data. It is possible to show that refitting the regression model after doing model selection with the lasso (or any other model selection method!) may lead to $\sqrt{n}$-biased estimates (which is one reason why a simple gaussian approximation will often fail for confidence intervals)

Fortunately, there has been much progress in recent years in developing inference methods that account for post-selection. Some relevant references for your case are: http://projecteuclid.org/euclid.aos/1460381681 and, https://arxiv.org/pdf/1602.07358.pdf. The techniques discussed in these references are implemented in the R package selectiveInference- https://cran.r-project.org/web/packages/selectiveInference/index.html. The selectiveInference package should produce the valid confidence intervals you need.

$\endgroup$
  • 1
    $\begingroup$ In the Machine Learning Specialization in Coursera by Univ. of Washington, the teachers of course 2 (Regression) devoted a whole week to Lasso regression. In one of the slides, the procedure I described (using Lasso to select features and then fitting a LS regression with only those variables) is denominated debiasing and is considered correct and illustrated with graphs from a paper by Mario Figueiredo. Check slide 105 here: github.com/MaxPoon/coursera-Machine-Learning-specialization/… $\endgroup$ – Pablo Jul 23 '17 at 17:18
  • $\begingroup$ While they recommend debiasing the lasso, they do not discuss hypothesis testing at all. Also, the term de-biasing is misleading, because while refitting the model gets rid of the downward bias induced by the lasso, it doesn't help with the upward bias induced by the winner's curse. As far as I know, the only way to truly de-bias the regression coefficient estimates of the selected model is to compute the conditional maximum likelihood estimates. arxiv.org/abs/1705.09417 $\endgroup$ – user3903581 Jul 24 '17 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.