# Inference after using Lasso for variable selection

I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no penalty. I'm doing this because I want unbiased estimates which Lasso cannot give me. I'd also like p-values and confidence intervals for the unbiased estimate.

I'm having trouble finding literature on this topic. Most of the literature I find is about putting confidence intervals on the Lasso estimates, not a refitted model.

From what I've read, simply refitting a model using the whole dataset leads to unrealistically small p-values/std errors. Right now, sample splitting (in the style of Wasserman and Roeder(2014) or Meinshausen et al. (2009)) seems to be a good course of action, but I'm looking for more suggestions.

Has anyone encountered this issue? If so, could you please provide some suggestions.

• I don't understand why should it should matter if the lasso estimator is biased as long as a the confidence intervals have (at least asymptotically) correct coverage. Is this the only reason why you want to fit OLS estimates on the support recovered by the lasso? – user795305 Jul 13 '17 at 17:02
• Maybe I've misunderstood what I've read, but doesn't the asymptotically correct coverage refer to biased estimate, not the true sparse-but-unbiased estimate? – Eli Jul 13 '17 at 17:10
• I'm not sure what you mean by "true sparse-but-unbiased" estimate, but if you know the lasso estimates have confidence intervals with asymptotically correct coverage, there shouldn't be more to do. The paper just linked by Greenparker (+1) is a really interesting one (and the most recent one that I know on this topic) that discusses (in part) how you could develop asymptotically correct confidence intervals on the lasso then ols coefficients. I'm trying to point out that you don't need to fit OLS in order to get unbiased coefficients, since unbiasedness doesn't matter. – user795305 Jul 13 '17 at 17:14
• I think I've been misunderstanding. The asymptotically correct coverage you're referring to is with respect to the true parameter. So even though Lasso gives biased coefficients, we can construct confidence intervals which have the correct coverage for the true parameter? – Eli Jul 13 '17 at 17:27
• SInce you've selected a model, you won't have unbased estimates if you estimate without Lasso. The coefficients of the terms in the model after select-variables-then-fit-via-OLS will actually be biased away from 0 (as with other forms of variable selection). A small amount of shrinkage may actually reduce the bias. – Glen_b Jul 14 '17 at 3:30

To add to the previous responses. You should definitely check out the recent work by Tibshirani and colleagues. They have developed a rigorous framework for inferring selection-corrected p-values and confidence intervals for lasso-type methods and also provide an R-package.

See:

Lee, Jason D., et al. "Exact post-selection inference, with application to the lasso." The Annals of Statistics 44.3 (2016): 907-927. (https://projecteuclid.org/euclid.aos/1460381681)

Taylor, Jonathan, and Robert J. Tibshirani. "Statistical learning and selective inference." Proceedings of the National Academy of Sciences 112.25 (2015): 7629-7634.

R-package:

https://cran.r-project.org/web/packages/selectiveInference/index.html

Generally, refitting using no penalty after having done variable selection via the Lasso is considered "cheating" since you have already looked at the data and the resulting p-values and confidence intervals are not valid in the usual sense.

This very recent paper looks at exactly what you want to do, and explains conditions under which fitting a lasso, choosing the important variables, and refitting without lasso penalty leads to valid $p$-values and confidence intervals. Their intuitive reasoning is that

the set of variables selected by the lasso is deterministic and non-data dependent with high probability.

Thus, peeking at the data twice is not a problem. You will need to see if for your problem the conditions stated in the paper hold or not.

(There are a lot of useful references in the paper as well)

Reference:

Zhao, S., Shojaie, A., & Witten, D. (2017). In defense of the indefensible: A very naive approach to high-dimensional inference. Retrieved from: https://arxiv.org/pdf/1705.05543.pdf

• +1 It is worth noting, however, that the authors explicitly do not recommend their approach except "in very large data settings": "We do not advocate applying the ... approach described above in most practical data analysis settings: we are confident that in practice ... this approach will perform poorly when the sample size is small or moderate, and/or the assumptions are not met" (at p. 27). For the record, this paper is Zhao, Shojaie, and Witten, In Defense of the Indefensible: A Very Naive Approach to High-Dimensional Inference (16 May 2017). – whuber Jul 13 '17 at 18:47
• @whuber And also keep in mind this paper is on arxiv.org - not sure if it's been peer-reviewed so there may be other issues with the author's methodology. – RobertF Aug 1 '19 at 18:26

I wanted to add some papers from the orthogonal/double machine learning literature that is becoming popular in the Applied Econometrics literature.

• Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. "Inference on treatment effects after selection among high-dimensional controls." The Review of Economic Studies 81.2 (2014): 608-650.

This paper addresses the theoretical properties of an OLS estimate of the effect of a variable after selecting the "other" controls using LASSO.

• Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, James Robins, Double/debiased machine learning for treatment and structural parameters, The Econometrics Journal, Volume 21, Issue 1, 1 February 2018, Pages C1–C68, https://doi.org/10.1111/ectj.12097

This develops the comprehensive theory for using a number of non-parametric methods (ML algorithms) to non-linearly control for a high-dimensional nuisance parameter (confounders) and then study the impact of a specific covariate on the outcome. They deal with partially-linear frameworks and completely parametric frameworks. They also consider situations where the variable of interest is confounded.