Overall Multiple Regression (F test) is not significant but individual regression coefficients are, when controlling for covariates/confounders

Question

I am interested in performing a multiple linear regression in order to determine if levels of my protein of interest is associated with cognitive decline in pre-symptomatic Alzheimer's disease. However, based on the literature I have decided to add several covariates/ confounders (such as age, gender, years of education and ApoE4 carrier status) to the model. When I run the analysis, the overall model (F-statistic from ANOVA table) is not significant (p = 0.069). From what I understand, this suggests that none of the regression coefficients differ from 0. However, when I look at the significance of the regression coefficient for my protein of interest, it is highly significant (p = 0.008) when adjusting for the covariates/confounders.

Is it possible that the non-significance of the overall model is due to the the non-significance of the regression coefficients of some of these confounders? For instance, the significance of the regression coefficients are: ApoE4 (p = 0.855), age (p = 0.180), gender (p = 0.085) education (p = 0.469).

I am only interested in the effects of my protein (p = 0.008) on my dependent variable (cognitive decline) while controlling for these covariates/ confounders. I am not necessarily interested in the individual regression coefficients of these covariates/ confounders.

Can I still say that there is a significant relationship between my protein and cognitive decline? Multicollinearity and heteroskedasticity do not appear to be present

Thank you

Answer 1

You might want to look at Frank Harrell's blog post about how to evaluate the value added by a new biomarker. What's most important at first is a test of whether the model with all the predictors including your protein of interest is better than a model with all the other predictors that you are including as controls but omitting your protein.

With least-squares regression models of class lm in R, that comparison can be done via the anova() function called with those two nested models as arguments. I suspect you will find your model with the protein to be superior in that test.

Then, however, you need to go on to evaluate just how much adding the protein to the model helps. As Harrell says:

When the outcome variable Y is continuous, there are only three measures of added value that are commonly used: increase in $R^2$, decrease in mean squared prediction error, and decrease in mean absolute prediction error.

So look at one or more of those measures, too. It's quite possible to have a "statistically significant" result that isn't of practical importance.

My guess is that your situation arose from having a fairly small number of cases relative to the number of predictors in your model. For categorical predictors, every category beyond the first counts as a predictor in this sense. If you categorized a continuous predictor like age, you run a risk of adding so many predictors that you lose a lot of power.

An alternate approach with too many control variables is to use a linear model with a ridge-regression penalty on the variables you are trying to control for, while keeping your main predictor of interest unpenalized. See Stat. Med. 2016 Nov 10;35(25):4546-4558.

That said, you should be wary of putting too much faith in results based on relatively small numbers of cases.

Finally, you don't provide many details on the model. I'm assuming the you've done appropriate quality control on the modeling. Without that, all interpretations of your results might be questionable.

Answer 2

From An Introduction to Statistical Learning, second edition (James et al, 2021):

Given these individual p-values for each variable, why do we need to look at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large.

For instance, consider an example in which p = 100 and H0 : β1 = β2 = ··· = βp = 0 is true, so no variable is truly associated with the response. In this situation, about 5 % of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0.05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response.8 In fact, it is likely that we will observe at least one p-value below 0.05 by chance! Hence, if we use the individual t-statistics and associated p-values in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship. However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H0 is true, there is only a 5 % chance that the F-statistic will result in a pvalue below 0.05, regardless of the number of predictors or the number of observations.

So, I would say that your model does not provide reliable results about the individual significance of protein. For further details, check chapter 3 of the aforementioned book.