Statistical inference in lasso - does data splitting work?

Question

There is something about statistical inference after (linear) model selection that I can’t wrap my head around.

Let’s take Lasso, for example; I am interested in doing hypothesis testing on the (estimated parameters of the) selected variables. Obviously, one needs to correct for the “data mining” that goes into searching for the optimal lambda.

The method in question is what some refer to as data splitting – you split your data in half, select the model using the first part, and use the second part to get the p-values. In my reading of the literature, academic researchers offer two conflicting views of this approach.

1. On one hand, there are the statisticians. They are totally fine with this approach. Wasserman & Roeder wrote a paper showing this works.
2. On the other hand, there are the econometricians. Leeb and Pötscher, in a series of papers say that we can't do that. It is best described by Stata here:

Everyone agrees that it would be better to split your data into two samples; use the first to select the model and use the second to refit it. You will now have the standard errors you need. But even this will not provide good estimates and standard errors if your interest is in the true model that generated the data. The problem is twofold...

More references are here (p.34, 1st para), and here.

What am I missing here? Why is this discrepancy between the two papers?

Note: This question is not about de-biasing the lasso, selective inference or other approaches to hypothesis testing after model selection.

Answer 1

It is not very clear what is meant with

if your interest is in the true model that generated the data

But I can imagine the following problem when the true model is (due to bias) not inside the space of all potential models:

• Residuals are often used as a measure of the random statistical variation in the model.

  However in Lasso or other methods with bias, the residuals are a measure of random statistical variation plus bias. Methods that assume that the residuals are purely random (e.g assume that they follow a multivariate normal distribution with zero mean) will compute p-values that make little sense.

  You can test maybe the null hypothesis, (all the coefficients are zero) as that model is inside the model space. But something like confidence intervals for the parameters is a different thing.