Should I determine whether an independent variable affects my data using regression parameter significance test or cross validation

Question

Let's say that I'm trying to understand the factors affecting heart rate and I have a series of 500 measurement timepoints from 100 people. I have a wild mixture of categorical and continuous independent variables (that are not particularly uncorrelated..) that could affect heart rate and I want to know if one particular independent variable affects heart-rate.

I thought of two ways of doing this but I am not sure which is more appropriate or whether I should expect them to yield similar results.

1. Fit a multiple regression model with all independent variables predicting heart rate and then do significance tests on regression coefficients.

2. Fit a multiple regression model with all independent variables predicting heart rate, then fit a multiple regression model using all variables except for my variable of interest and do calculate both of these models ability to predict unseen data.

I've been in an analogous situation multiple times, where the two methods have yielded different results such that method 1 tells me an independent variable is affecting my data while method 2 tells me it is not. My intuition is to go with what method 2 is saying but have no real rational for doing so.

My questions are:

1. Under what conditions should each method be used?
2. What literature is relevant to this question?
3. How are cross-validation and hypothesis tests (formally) related?

Answer 1

The comment from @gung is directly on point. Your two methods are measuring different things.

Method 1 is, as noted in the answer from @AlexandreCazenave-Lacroutz, "the classical way to make inference." But think closely about what Method 1 is testing. It's whether your variable of interest is "significantly" related to heart rate in a very specific sense: if there were really no relationship and you repeated the experiment (with the same number of cases) multiple times, you would find such a large relationship by chance in less than 1 out of 20 experiments (at p < 0.05). With small sample sizes and particularly with correlations among predictor variables, it's quite possible for Method 1 to miss a true relationship between your variable and heart rate; a larger sample size might let you document its "significance."

Method 2 has to do with predictive performance. It's quite possible--even likely--that a variable that doesn't pass the strict test of Method 1 could still improve performance when added to other predictor variables. For predictive modeling it's generally considered good practice to include all variables that are reasonably related to outcome provided that you are not overfitting. For example, Harrell in Regression Modeling Strategies lists as one recommended approach in this context (page 89):

fitting fully pre-specified models without deletion of “insignificant” predictors.

In summary, Method 2 can document whether your variable bears some relationship to heart rate, in that including information about it provides a better estimate of heart rate. Method 1 documents whether the combination of the magnitude of that relationship with the size of your data set is sufficient to rule out a chance relationship at a specified level of significance. Which is "more appropriate" depends on which of those criteria is more important to you.

In response to comment:

You need to make sure that your model comparison test in Method 2 has sufficient sensitivity to see differences. For example, Harrell explains (see the links from that page for more details):

The two most commonly used resampling methods are cross-validation and bootstrapping. To be as good as the bootstrap, about 100 repeats of 10-fold cross-validation are required.

Note that bootstrapping (sampling with replacement) is more closely akin to re-sampling from the original population than is cross-validation (a form of sub-sampling without replacement). See this page for some justification. I typically use cross-validation for things like choosing penalty values in ridge regression or LASSO, then use multiple bootstrap samples to repeat the model-building process, with performance testing of the multiple models on the original data set. That's a good way to estimate bias, "optimism" in estimates due to overfitting, and how well your modeling process would be likely to work on a new sample from the original population. See this page for an outline of this approach and links to further documentation.

Answer 2

Let us assume that your variable of interest has a very small but significant impact. Maybe it is small compared to the noise in your out-of-sample validation data (and withdrawing the variable of interest might have an impact of the other coefficients - by chance it can improve your out-of-sample fit). You would have : method 1 tells me an independent variable is affecting my data while method 2 tells me it is not. Method 2 would dismiss it whereas it does have an impact.

Method 1 is the classical way to make inference regarding correlations: with the hypotheses behind your model, the variation behind your point estimate is the confidence interval. And, following the model that you estimate anyway, under the null-hypothesis, there is a p-value than you would have had this point estimates. If it is low enough, you reject the hypothesis that there is no effect.

Note however that it is an impact mutadis mutandis - the other variables in the model being held constant. This should not be forgotten in the interpretation.