2
$\begingroup$

I am interested in performing a multiple linear regression in order to determine if levels of my protein of interest is associated with cognitive decline in pre-symptomatic Alzheimer's disease. However, based on the literature I have decided to add several covariates/ confounders (such as age, gender, years of education and ApoE4 carrier status) to the model. When I run the analysis, the overall model (F-statistic from ANOVA table) is not significant (p = 0.069). From what I understand, this suggests that none of the regression coefficients differ from 0. However, when I look at the significance of the regression coefficient for my protein of interest, it is highly significant (p = 0.008) when adjusting for the covariates/confounders.

Is it possible that the non-significance of the overall model is due to the the non-significance of the regression coefficients of some of these confounders? For instance, the significance of the regression coefficients are: ApoE4 (p = 0.855), age (p = 0.180), gender (p = 0.085) education (p = 0.469).

I am only interested in the effects of my protein (p = 0.008) on my dependent variable (cognitive decline) while controlling for these covariates/ confounders. I am not necessarily interested in the individual regression coefficients of these covariates/ confounders.

Can I still say that there is a significant relationship between my protein and cognitive decline? Multicollinearity and heteroskedasticity do not appear to be present

Thank you

$\endgroup$
5
  • $\begingroup$ Can I ask what is the purpose of the study? For instance, are you 1) evaluating whether your protein could be a biomarker useful for clinical prediction of disease? Or 2) testing whether your biomarker is associated with disease to reach some biological or causal insight? $\endgroup$
    – Lachlan
    Jan 24 at 0:28
  • $\begingroup$ Hi Lachlan, it is #2 I am testing whether or not is has a causal role during the pre-symptomatic stage of the disease. $\endgroup$
    – MQ99
    Jan 24 at 2:14
  • $\begingroup$ I believe @EdM below may have been under the impression that #1 was true. In this case, imo, your model should be driven by an understanding of the causal system underlying the disease, rather than any statistical criteria at all. See the paper 'Principles of Confounder Selection' by Tyler VanderWeele. After you have built an appropriate causal model (ideally based on a directed acyclic graph), you can then interpret the coefficients + p values from your biomarker variable and ignore the p value for the overall model. $\endgroup$
    – Lachlan
    Jan 24 at 9:10
  • $\begingroup$ @Lachlan, from what you suggest, is it still possible to interpret R squared and adjusted R squared? $\endgroup$
    – MQ99
    Jan 31 at 21:48
  • $\begingroup$ No, I don't really think so. At least not with much meaning. If you want to quantify added predictive value of the biomarker (beyond the other variables in your model) the Frank Harrell link in EdM's post below will be helpful. Alternatively, you could simply calculate the R squared difference between a model including the confounders, and one additionally including the biomarker. $\endgroup$
    – Lachlan
    Feb 1 at 7:30

2 Answers 2

1
$\begingroup$

You might want to look at Frank Harrell's blog post about how to evaluate the value added by a new biomarker. What's most important at first is a test of whether the model with all the predictors including your protein of interest is better than a model with all the other predictors that you are including as controls but omitting your protein.

With least-squares regression models of class lm in R, that comparison can be done via the anova() function called with those two nested models as arguments. I suspect you will find your model with the protein to be superior in that test.

Then, however, you need to go on to evaluate just how much adding the protein to the model helps. As Harrell says:

When the outcome variable Y is continuous, there are only three measures of added value that are commonly used: increase in $R^2$, decrease in mean squared prediction error, and decrease in mean absolute prediction error.

So look at one or more of those measures, too. It's quite possible to have a "statistically significant" result that isn't of practical importance.

My guess is that your situation arose from having a fairly small number of cases relative to the number of predictors in your model. For categorical predictors, every category beyond the first counts as a predictor in this sense. If you categorized a continuous predictor like age, you run a risk of adding so many predictors that you lose a lot of power.

An alternate approach with too many control variables is to use a linear model with a ridge-regression penalty on the variables you are trying to control for, while keeping your main predictor of interest unpenalized. See Stat. Med. 2016 Nov 10;35(25):4546-4558.

That said, you should be wary of putting too much faith in results based on relatively small numbers of cases.

Finally, you don't provide many details on the model. I'm assuming the you've done appropriate quality control on the modeling. Without that, all interpretations of your results might be questionable.

$\endgroup$
3
  • $\begingroup$ Hi EdM, in my analyses I verified that the model assumptions were satisfied. However, I do not have any "cases" (as in case-control) in my model. The subjects are all cognitively unaffected (healthy) individuals who are at a higher risk of developing the disease. $\endgroup$
    – MQ99
    Jan 26 at 1:49
  • $\begingroup$ @MQ99 by the “number of cases” I didn’t mean to imply a case-control situation, just the total number of participants/observations. Sorry for that terminological confusion. $\endgroup$
    – EdM
    Jan 26 at 3:55
  • $\begingroup$ thanks for the clarification $\endgroup$
    – MQ99
    Jan 31 at 21:49
0
$\begingroup$

From An Introduction to Statistical Learning, second edition (James et al, 2021):

Given these individual p-values for each variable, why do we need to look at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large.

For instance, consider an example in which p = 100 and H0 : β1 = β2 = ··· = βp = 0 is true, so no variable is truly associated with the response. In this situation, about 5 % of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0.05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response.8 In fact, it is likely that we will observe at least one p-value below 0.05 by chance! Hence, if we use the individual t-statistics and associated p-values in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship. However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H0 is true, there is only a 5 % chance that the F-statistic will result in a pvalue below 0.05, regardless of the number of predictors or the number of observations.

So, I would say that your model does not provide reliable results about the individual significance of protein. For further details, check chapter 3 of the aforementioned book.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.