Significance testing or cross validation?

Question

Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?

Answer 1

First, lets be explicit and put the question into the context of multiple linear regression where we regress a response variable, $y$, on several different variables $x_1, \ldots, x_p$ (correlated or not), with parameter vector $\beta = (\beta_0, \beta_1, \ldots, \beta_p)$ and regression function
$$f(x_1, \ldots, x_p) = \beta_0 + \beta_1 x_1 + \ldots + \beta_p x_p,$$ which could be a model of the mean of the response variable for a given observation of $x_1, \ldots, x_p$.

The question is how to select a subset of the $\beta_i$'s to be non-zero, and, in particular, a comparison of significance testing versus cross validation.

To be crystal clear about the terminology, significance testing is a general concept, which is carried out differently in different contexts. It depends, for instance, on the choice of a test statistic. Cross validation is really an algorithm for estimation of the expected generalization error, which is the important general concept, and which depends on the choice of a loss function.

The expected generalization error is a little technical to define formally, but in words it is the expected loss of a fitted model when used for prediction on an independent data set, where expectation is over the data used for the estimation as well as the independent data set used for prediction.

To make a reasonable comparison lets focus on whether $\beta_1$ could be taken equal to 0 or not.

• For significance testing of the null hypothesis that $\beta_1 = 0$ the main procedure is to compute a $p$-value, which is the probability that the chosen test-statistic is larger than observed for our data set under the null hypothesis, that is, when assuming that $\beta_1 = 0$. The interpretation is that a small $p$-value is evidence against the null hypothesis. There are commonly used rules for what "small" means in an absolute sense such as the famous 0.05 or 0.01 significance levels.
• For the expected generalization error we compute, perhaps using cross-validation, an estimate of the expected generalization error under the assumption that $\beta_1 = 0$. This quantity tells us how well models fitted by the method we use, and with $\beta_1 = 0$, will perform on average when used for prediction on independent data. A large expected generalization error is bad, but there are no rules in terms of its absolute value on how large it needs to be to be bad. We will have to estimate the expected generalization error for the model where $\beta_1$ is allowed to be different from 0 as well, and then we can compare the two estimated errors. Whichever is the smallest corresponds to the model we choose.

Using significance testing we are not directly concerned with the "performance" of the model under the null hypothesis versus other models, but we are concerned with documenting that the null is wrong. This makes most sense (to me) in a confirmatory setup where the main objective is to confirm and document an a priory well specified scientific hypothesis, which can be formulated as $\beta_1 \neq 0$.

The expected generalization error is, on the other hand, only concerned with average "performance" in terms of expected prediction loss, and concluding that it is best to allow $\beta_1$ to be different from 0 in terms of prediction is not an attempt to document that $\beta_1$ is "really" different from 0 $-$ whatever that means.

I have personally never worked on a problem where I formally needed significance testing, yet $p$-values find their way into my work and do provide sensible guides and first impressions for variable selection. I am, however, mostly using penalization methods like lasso in combination with the generalization error for any formal model selection, and I am slowly trying to suppress my inclination to even compute $p$-values.

For exploratory analysis I see no argument in favor of significance testing and $p$-values, and I will definitely recommend focusing on a concept like expected generalization error for variable selection. In other contexts where one might consider using a $p$-value for documenting that $\beta_1$ is not 0, I would say that it is almost always a better idea to report an estimate of $\beta_1$ and a confidence interval instead.

Answer 2

Simply using significance tests and a stepwise procedure to perform model selection can lead you to believe that you have a very strong model with significant predictors when you, in fact, do not; you may get strong correlations by chance and these correlations can seemingly be enhanced as you remove other unnecessary predictors.

The selection procedure, of course, keeps only those variables with the strongest correlations with the outcome and, as the stepwise procedure moves forward, the probability of committing a Type I error becomes larger than you would imagine. This is because the standard errors (and thus p-values) are not adjusted to take into account the fact that the variables were not selected for inclusion in the model randomly and multiple hypothesis tests were conducted to choose that set.

David Freedman has a cute paper in which he demonstrates these points called "A Note on Screening Regression Equations." The abstract:

Consider developing a regression model in a context where substantive theory is weak. To focus on an extreme case, suppose that in fact there is no relationship between the dependent variable and the explanatory variables. Even so, if there are many explanatory variables, the $R^2$ will be high. If explanatory variables with small t statistics are dropped and the equation refitted, the $R^2$ will stay high and the overall F will become highly significant. This is demonstrated by simulation and by asymptotic calculation.

One potential solution to this problem, as you mentioned, is using a variant of cross validation. When I don't have a good economic (my area of research) or statistical reason to believe my model, this is my preferred approach to selecting an appropriate model and performing inference.

Other respondents might mention that stepwise procedures using the AIC or BIC are asympotically equivalent to cross validation. This only works as the number of observations relative to the number of predictors gets large, however. In the context of having many variables relative to the number of observations (Freedman says 1 variable per 10 or fewer observations), selection in this manner can exhibit the poor properties discussed above.

In an age of powerful computers, I don't see any reason not to use cross validation as a model selection procedure over stepwise selection.