2
$\begingroup$

There is something about statistical inference after (linear) model selection that I can’t wrap my head around.

Let’s take Lasso, for example; I am interested in doing hypothesis testing on the (estimated parameters of the) selected variables. Obviously, one needs to correct for the “data mining” that goes into searching for the optimal lambda.

The method in question is what some refer to as data splitting – you split your data in half, select the model using the first part, and use the second part to get the p-values. In my reading of the literature, academic researchers offer two conflicting views of this approach.

  1. On one hand, there are the statisticians. They are totally fine with this approach. Wasserman & Roeder wrote a paper showing this works.
  2. On the other hand, there are the econometricians. Leeb and Pötscher, in a series of papers say that we can't do that. It is best described by Stata here:

Everyone agrees that it would be better to split your data into two samples; use the first to select the model and use the second to refit it. You will now have the standard errors you need. But even this will not provide good estimates and standard errors if your interest is in the true model that generated the data. The problem is twofold...

More references are here (p.34, 1st para), and here.

What am I missing here? Why is this discrepancy between the two papers?

Note: This question is not about de-biasing the lasso, selective inference or other approaches to hypothesis testing after model selection.

$\endgroup$
4
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Oct 18, 2022 at 23:13
  • 1
    $\begingroup$ Do you remember where in Leeb and Pötscher's paper they discuss sample splitting? Searches for "split", "half" and "subsample" yield nothing in that paper. $\endgroup$ Oct 19, 2022 at 6:47
  • $\begingroup$ Splitting your data in halfs is great when there is an abundance of data but much in statistics becomes easy when samples are large. Most of the time data is scarce and your chances of finding the 'right' model in half of your data are even smaller then finding it in all of your data. Those are truth independent of your denomination but will lead to different conclusions depending on the problems you work on. See 'cross validation' for an approach to reduce the problem of scarce data in this context. $\endgroup$
    – Bernhard
    Oct 19, 2022 at 8:23
  • $\begingroup$ @RichardHardy, great point. I dug into this a bit. Chernozhukov et al. p.34 para 1 say Leeb and Potscher discuss this in their 2008 papers (not the earlier and more cited ones). In short, it seems like Lasso can make mistakes by not including variables with small coefficients, so we need to account for this. In any given sample, such a mistake is possible, so data split is unreliable. This makes sense intuitively. So my current best guess is that statisticians assume this away while econometricians work through it. $\endgroup$ Oct 20, 2022 at 21:19

1 Answer 1

0
$\begingroup$

It is not very clear what is meant with

if your interest is in the true model that generated the data

But I can imagine the following problem when the true model is (due to bias) not inside the space of all potential models:

  • Residuals are often used as a measure of the random statistical variation in the model.

    However in Lasso or other methods with bias, the residuals are a measure of random statistical variation plus bias. Methods that assume that the residuals are purely random (e.g assume that they follow a multivariate normal distribution with zero mean) will compute p-values that make little sense.

    You can test maybe the null hypothesis, (all the coefficients are zero) as that model is inside the model space. But something like confidence intervals for the parameters is a different thing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.