Lasso and statistical signficance of selected variables

Question

I'm looking at a regression model where a very large number of possible explanatory variables are being evaluated, and a small number are finally chosen via the lasso method of variable selection. The $\lambda$ tuning parameter in the lasso is chosen by looking at cross-validation forecast performance, which is pretty standard.

However, when I take the list of chosen variables and just run OLS on them, many turn out to be statistically insignificant. That may be perfectly fine if they are jointly significant and the forecast performance is superior to other models (in addition, there would be a question of what the t-test means when you have already screened the variables in a separate step, but I'm leaving that aside).

I'm curious though whether it makes sense to look at statistical significance of individual variables in a model chosen by lasso using CV forecast performance to select the tuning parameter. The problem is that lasso ends up selecting various dummy variables that are only true on small segments of the population and which are insignificant in OLS, and there is a natural question as to whether the model should be judgmentally simplified.

Answer 1

There are at least two things to consider here.

First, it's important to realize that the p-values in a regression make quite a few assumptions in order to be valid. Most important for your case, they assume you followed a procedure like this:

I collected data, and decided what model to fit without looking at the data I collected. Then I fit my pre-determined model, which I assume fits the data well, without really checking and making any changes.

Under these assumptions, the p-values are meaningful. If you make changes to your model based on the data you collected, variable selection using the LASSO for example, the p-values estimated from a linear model are not meaningful. This part of the question may be addressed by user2530062's answer to this question, given that p-values are actually of interest to you.

Secondly, there is the question of what question you are attempting to answer. The p-values address a very specific question:

Under the assumption that this model is correct for the data I am collecting, and that the true value of this parameter I am interested in estimating is in reality zero, what is the probability that I would observe an equally or more extreme value of the estimated parameter when I fit my model to a sample of data collected from this process.

If that's the question that you are interested in answer, then carefully constructing your model so that the p-value is valid is how to go about it. But I suspect this may not be the question you are actually interested in answering. Maybe your question is more like this:

What is the probability that including this parameter in the model improves the predictive accuracy of my model for this process?

A p-value does not give you any real information on that question, or the infinity of other questions that p-values were not designed to address. Instead, you should design a procedure to measure exactly the thing you are interested in. In the above example, a rigorous procedure using the bootstrap to estimate the probability that including the parameter in the model improved predictive accuracy, along with cross validation to estimate the regularization parameter, would do you well.

Answer 2

This paper tries to provide approach to calculate p-values in elasticnet. I have been struggling to find time to implement it, as it appears to be experimental and not included in any official R package.

http://statweb.stanford.edu/~tibs/ftp/covtest.pdf

It does not answer the theoretical part of your question, but may bring you closer to an answer if you calculate p-values for elasticnet.