There is something about statistical inference after (linear) model selection that I can’t wrap my head around.
Let’s take Lasso, for example; I am interested in doing hypothesis testing on the (estimated parameters of the) selected variables. Obviously, one needs to correct for the “data mining” that goes into searching for the optimal lambda.
The method in question is what some refer to as data splitting – you split your data in half, select the model using the first part, and use the second part to get the p-values. In my reading of the literature, academic researchers offer two conflicting views of this approach.
- On one hand, there are the statisticians. They are totally fine with this approach. Wasserman & Roeder wrote a paper showing this works.
- On the other hand, there are the econometricians. Leeb and Pötscher, in a series of papers say that we can't do that. It is best described by Stata here:
Everyone agrees that it would be better to split your data into two samples; use the first to select the model and use the second to refit it. You will now have the standard errors you need. But even this will not provide good estimates and standard errors if your interest is in the true model that generated the data. The problem is twofold...
More references are here (p.34, 1st para), and here.
What am I missing here? Why is this discrepancy between the two papers?
Note: This question is not about de-biasing the lasso, selective inference or other approaches to hypothesis testing after model selection.