# Should I determine whether an independent variable affects my data using regression parameter significance test or cross validation

Let's say that I'm trying to understand the factors affecting heart rate and I have a series of 500 measurement timepoints from 100 people. I have a wild mixture of categorical and continuous independent variables (that are not particularly uncorrelated..) that could affect heart rate and I want to know if one particular independent variable affects heart-rate.

I thought of two ways of doing this but I am not sure which is more appropriate or whether I should expect them to yield similar results.

1. Fit a multiple regression model with all independent variables predicting heart rate and then do significance tests on regression coefficients.

2. Fit a multiple regression model with all independent variables predicting heart rate, then fit a multiple regression model using all variables except for my variable of interest and do calculate both of these models ability to predict unseen data.

I've been in an analogous situation multiple times, where the two methods have yielded different results such that method 1 tells me an independent variable is affecting my data while method 2 tells me it is not. My intuition is to go with what method 2 is saying but have no real rational for doing so.

My questions are:

1. Under what conditions should each method be used?
2. What literature is relevant to this question?
3. How are cross-validation and hypothesis tests (formally) related?
• The question seems to confound (a) 1 model vs 2, w/ (b) statistical tests vs. predictive ability. Which contrast are you interested in? Aug 30, 2019 at 18:55
• I am interested in drawing inferences about whether my variable of interest affects heart rate. I am not interested in predictive ability. I can intuitively see how these things may be different but surely there must be pretty deep connections between statistical tests and predictive ability? Why can they be so easily separated? Aug 30, 2019 at 19:20

The comment from @gung is directly on point. Your two methods are measuring different things.

Method 1 is, as noted in the answer from @AlexandreCazenave-Lacroutz, "the classical way to make inference." But think closely about what Method 1 is testing. It's whether your variable of interest is "significantly" related to heart rate in a very specific sense: if there were really no relationship and you repeated the experiment (with the same number of cases) multiple times, you would find such a large relationship by chance in less than 1 out of 20 experiments (at p < 0.05). With small sample sizes and particularly with correlations among predictor variables, it's quite possible for Method 1 to miss a true relationship between your variable and heart rate; a larger sample size might let you document its "significance."

Method 2 has to do with predictive performance. It's quite possible--even likely--that a variable that doesn't pass the strict test of Method 1 could still improve performance when added to other predictor variables. For predictive modeling it's generally considered good practice to include all variables that are reasonably related to outcome provided that you are not overfitting. For example, Harrell in Regression Modeling Strategies lists as one recommended approach in this context (page 89):

fitting fully pre-specified models without deletion of “insignificant” predictors.

In summary, Method 2 can document whether your variable bears some relationship to heart rate, in that including information about it provides a better estimate of heart rate. Method 1 documents whether the combination of the magnitude of that relationship with the size of your data set is sufficient to rule out a chance relationship at a specified level of significance. Which is "more appropriate" depends on which of those criteria is more important to you.

In response to comment:

You need to make sure that your model comparison test in Method 2 has sufficient sensitivity to see differences. For example, Harrell explains (see the links from that page for more details):

The two most commonly used resampling methods are cross-validation and bootstrapping. To be as good as the bootstrap, about 100 repeats of 10-fold cross-validation are required.

Note that bootstrapping (sampling with replacement) is more closely akin to re-sampling from the original population than is cross-validation (a form of sub-sampling without replacement). See this page for some justification. I typically use cross-validation for things like choosing penalty values in ridge regression or LASSO, then use multiple bootstrap samples to repeat the model-building process, with performance testing of the multiple models on the original data set. That's a good way to estimate bias, "optimism" in estimates due to overfitting, and how well your modeling process would be likely to work on a new sample from the original population. See this page for an outline of this approach and links to further documentation.

• thanks for your answer! There are two things I'm confused about. Firstly, it seems like you are suggesting that Method 2 is likely to be more 'lenient'. However, what I have been observing is the opposite, that Method 1 is more 'lenient'. Secondly, as I understand it, the idea behind inferential statistics is that if you repeat the experiment again, you should see the same result (obviously simplifying here...). However, this seems very closely aligned with the idea of cross-validation, and I don't understand how failing 'Method 2' doesn't suggest that the finding will not replicate? Sep 3, 2019 at 15:13
• presumably things like overfitting come into play here, but I'm not sure how this all fits together Sep 3, 2019 at 15:29
• @user3235916 bootstrapping may be better related to "repeat[ing] the experiment again" than is cross validation. Much also depends on the details of how you do the cross validation. A single cross-validation might not have the power to detect a true difference in predictive ability between 2 models. See this page for some discussion.
– EdM
Sep 3, 2019 at 15:53
• thanks a bunch! This is exactly what I was looking for :) Sep 5, 2019 at 9:35

Let us assume that your variable of interest has a very small but significant impact. Maybe it is small compared to the noise in your out-of-sample validation data (and withdrawing the variable of interest might have an impact of the other coefficients - by chance it can improve your out-of-sample fit). You would have : method 1 tells me an independent variable is affecting my data while method 2 tells me it is not. Method 2 would dismiss it whereas it does have an impact.

Method 1 is the classical way to make inference regarding correlations: with the hypotheses behind your model, the variation behind your point estimate is the confidence interval. And, following the model that you estimate anyway, under the null-hypothesis, there is a p-value than you would have had this point estimates. If it is low enough, you reject the hypothesis that there is no effect.

Note however that it is an impact mutadis mutandis - the other variables in the model being held constant. This should not be forgotten in the interpretation.