Inference after using Lasso for variable selection

Question

I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no penalty. I'm doing this because I want unbiased estimates which Lasso cannot give me. I'd also like p-values and confidence intervals for the unbiased estimate.

I'm having trouble finding literature on this topic. Most of the literature I find is about putting confidence intervals on the Lasso estimates, not a refitted model.

From what I've read, simply refitting a model using the whole dataset leads to unrealistically small p-values/std errors. Right now, sample splitting (in the style of Wasserman and Roeder(2014) or Meinshausen et al. (2009)) seems to be a good course of action, but I'm looking for more suggestions.

Has anyone encountered this issue? If so, could you please provide some suggestions.

Answer 1

To add to the previous responses. You should definitely check out the recent work by Tibshirani and colleagues. They have developed a rigorous framework for inferring selection-corrected p-values and confidence intervals for lasso-type methods and also provide an R-package.

See:

Lee, Jason D., et al. "Exact post-selection inference, with application to the lasso." The Annals of Statistics 44.3 (2016): 907-927. (https://projecteuclid.org/euclid.aos/1460381681)

Taylor, Jonathan, and Robert J. Tibshirani. "Statistical learning and selective inference." Proceedings of the National Academy of Sciences 112.25 (2015): 7629-7634.

R-package:

https://cran.r-project.org/web/packages/selectiveInference/index.html

Answer 2

Generally, refitting using no penalty after having done variable selection via the Lasso is considered "cheating" since you have already looked at the data and the resulting p-values and confidence intervals are not valid in the usual sense.

This very recent paper looks at exactly what you want to do, and explains conditions under which fitting a lasso, choosing the important variables, and refitting without lasso penalty leads to valid $p$-values and confidence intervals. Their intuitive reasoning is that

the set of variables selected by the lasso is deterministic and non-data dependent with high probability.

Thus, peeking at the data twice is not a problem. You will need to see if for your problem the conditions stated in the paper hold or not.

(There are a lot of useful references in the paper as well)

---

Reference:

Zhao, S., Shojaie, A., & Witten, D. (2017). In defense of the indefensible: A very naive approach to high-dimensional inference. Retrieved from: https://arxiv.org/pdf/1705.05543.pdf

Answer 3

I wanted to add some papers from the orthogonal/double machine learning literature that is becoming popular in the Applied Econometrics literature.

• Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. "Inference on treatment effects after selection among high-dimensional controls." The Review of Economic Studies 81.2 (2014): 608-650.

  This paper addresses the theoretical properties of an OLS estimate of the effect of a variable after selecting the "other" controls using LASSO.

• Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, James Robins, Double/debiased machine learning for treatment and structural parameters, The Econometrics Journal, Volume 21, Issue 1, 1 February 2018, Pages C1–C68, https://doi.org/10.1111/ectj.12097

  This develops the comprehensive theory for using a number of non-parametric methods (ML algorithms) to non-linearly control for a high-dimensional nuisance parameter (confounders) and then study the impact of a specific covariate on the outcome. They deal with partially-linear frameworks and completely parametric frameworks. They also consider situations where the variable of interest is confounded.