2
$\begingroup$

In the book ESL, I read about model selection methods involving choosing a subset of the predictors. However, the books says that the standard errors are not valid since they do not account for the search process. Rather, we should use bootstrap/cross validation for this purpose.

I don't understand how the search process invalidates the standard errors of the model. And if this concept can be generalized, I am hoping for an explanation of how standard errors accumulate through some multi-step modelling process.

$\endgroup$
  • $\begingroup$ Was the explanation below enough ? $\endgroup$ – keepAlive Sep 14 at 21:05
2
$\begingroup$

Let's look at the standard linear regression set-up. We have $Y = X\beta + \varepsilon$ where $X$ is fixed, $\beta \in \mathbb R^p$ is fixed but unknown, and $\varepsilon \sim \mathcal N(0, \sigma^2 I)$ for some fixed but unknown $\sigma^2 > 0$. We'll also assume that $X$ is full rank so that the standard OLS estimator of $\beta$ is uniquely defined.

If we fit the full model we get $\hat \beta \big\vert \{X, \sigma^2, \beta\} \sim \mathcal N(\beta, \sigma^2 (X^TX)^{-1})$ where I'm explicitly showing the dependence on $X$ and the parameters. Now let's say we compute all $p$ models with $p-1$ predictors and $\hat \beta_{(p-1)}$ is the best one (so we've effectively done one step in a backward selection). Let $X_{(p-1)}$ be the corresponding design matrix.

If we never even saw all of $X$ and $X_{(p-1)}$ was what we started with, then it would indeed be the case that (under the same modeling assumptions) $$\hat \beta_{(p-1)}\big\vert \{\beta_{(p-1)}, \sigma^2, X_{(p-1)}\} \sim \mathcal N(\beta_{(p-1)}, \sigma^2 (X_{(p-1)}^TX_{(p-1)})^{-1}).$$

But we aren't just given $X_{(p-1)}$. We arrive at it by some procedure which also considers the random data $Y$ and the now-deleted column (say $i$) of $X$. So if we've done some feature selection we now are considering something of the form $$ \hat \beta_{(p-1)}\big\vert \{Y, \beta, \sigma^2, X\} $$

which is conditioned on the response $Y$ (in some perhaps very complicated way) and is a very different beast from $\hat \beta_{(p-1)}\big\vert \{\beta_{(p-1)}, \sigma^2, X_{(p-1)}\}$. So we can see that the actual statistics that we are computing in our feature selection (FS) simply don't have the distribution that they ostensibly have, so SEs computed using that assumed distribution have no reason to be correct. And since many FS procedures involve choosing the columns that best explain the response we'll typically find that the naive SEs are way too small.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.