4
$\begingroup$

These predictors are highly correlated.

In another (significant) model, when the predictors are ran separately, one of them accounted for higher unique variance (around 40%) than the variance accounted for when they were entered together (around 10%). I am also wondering why that is?

$\endgroup$
4
  • $\begingroup$ It this a hypothetical question or the proportions of variance explained you cite come from an actual example? If it's the second case, please provide more information about the data, the models and the two predictors. $\endgroup$
    – dipetkov
    Oct 25, 2022 at 5:21
  • $\begingroup$ @dipetkov My first question is hypothetical, the second question is from a study I have read (full text here: onlinelibrary.wiley.com/doi/full/10.1111/…) note that how when they included the two SES factors (parent education and income-to-needs ratio) together in the hippocampus model, the R^2 change is 12.4% (see Table 2). When they entered the two predictors separately, income-to-needs ratio alone accounted for 44% of unique varaince. A simialr drastic increase is seen in the amygdala model. $\endgroup$
    – Yvette
    Oct 26, 2022 at 18:24
  • $\begingroup$ @dipetkov This same study made me wonder why they did not entered the two variables seperately to begin with, and whether an insignificant overall multple-regression model could become significant if some of its predictors were entered seperately. In the latter case, I'm also wondering if they should adjust the alpha value for each model. $\endgroup$
    – Yvette
    Oct 26, 2022 at 18:28
  • $\begingroup$ I finally added the promised simulation. $\endgroup$
    – Dave
    Mar 22, 2023 at 19:22

2 Answers 2

4
$\begingroup$

TLDR: This question is inspired by the $R^2$ numbers reported in an article about the effects of socioeconomic status (SES) on brain development in children. It turns out that some $R^2$s are incorrectly reported: the typos are not in the main results table and don't change any of the conclusions but there is an apparent inconsistency which the OP noticed.

I contacted the article's first author, Prof. Noble who is at Columbia University, and she was kind enough to look at the data records and reply. She confirmed there is a typo in the $R^2$ change numbers reported.

Instead of 0.59, the $R^2$ change should have been reported as 0.059. The Beta and p-value are correct. So the sentence should read as follows:

"In the amygdala, when controlling for age, total cortical volume, and gender, parent education alone accounted for significant unique variance, with lower education levels associated with larger amygdala size (R2 change = 0.059, Beta = 0.286, p < .002; see Figure 2)."


As @Dave explains (+1), it's theoretically possible to add noisy variables which nevertheless use up degrees of freedom, so that the bigger model is "worse" than the smaller model. This is called overfitting.

I'm more interested in the second part of the OP's question. I paraphrase:

In a model Y ~ covariates + X1, the predictor X1 accounts for 40% of the variance in Y. But in the model Y ~ covariates + X1 + X2, the two predictors X1 and X2 together account for only 10% of the variance in Y.

The OP explains in a comment that these numbers come from the following study of the effects of socioeconomic status (SES) on brain development in children:

Noble, K.G., Houston, S.M., Kan, E. and Sowell, E.R. (2012), Neural correlates of socioeconomic status in the developing human brain. Developmental Science, 15: 516-527. https://doi.org/10.1111/j.1467-7687.2012.01147.x

I suspect that the numbers are mistakenly reported by Noble et al. Or perhaps I don't follow their calculations. In any case, here are the numbers which don't seem to make sense. I combine data from Table 2 and the text below Table 2 on page 6.

brain region (Y) regression R-squared
amygdala Y ~ age + volume + gender 0.634
Y ~ age + volume + gender + income 0.634 + 0.001
Y ~ age + volume + gender + education 0.634 + 0.59
Y ~ age + volume + gender + education + income 0.634 + 0.088
hippocampus Y ~ age + volume + gender 0.457
Y ~ age + volume + gender + income 0.457 + 0.44
Y ~ age + volume + gender + education 0.457 + 0.02
Y ~ age + volume + gender + education + income 0.457 + 0.124

Note that one R-squared ends up greater than one: 0.634 + 0.59 = 1.224. This is impossible.

So I suggest that both the 0.59 and the 0.44 are mis-reported. It seems likely that the correct $R^2$ change from the baseline step-1 regression are 0.059 and 0.044, for the amygdala and the hippocampus respectively. Those numbers would be consistent with the full models Y ~ covariates + eduction + income because the $R^2$ changes from adding each predictor separately add up to at most the combined $R^2$ change, as expected: 0.001 + 0.059 ≤ 0.088 and 0.044 + 0.02 ≤ 0.124.

$\endgroup$
5
  • $\begingroup$ This is a well-researched post. Why didn't it get any votes? +1. $\endgroup$ Nov 7, 2022 at 5:42
  • $\begingroup$ @User1865345 Thanks. It's probably that the OP hasn't returned to the question (yet). It happens. $\endgroup$
    – dipetkov
    Nov 7, 2022 at 10:22
  • 1
    $\begingroup$ I didn't say specifically keeping in mind whether OP visited again or not, which would have been better if they could see the post. What I said was noting it didn't garner any upvote against the well-researched work you did. Appreciated the effort from your part and that's why the upvote. $\endgroup$ Nov 7, 2022 at 10:31
  • $\begingroup$ Hey @dipetkov, thanks so much for your response! I got super busy last semester but did saw your reply at a later time. I discussed it with a few of my profs and they did let me know that sometimes it can be easy to make minor mistakes in a paper and get confused with your own data and not have it noticed during the peer review process. This has been a learning experience for me and I will read more on overfitting as well! $\endgroup$
    – Yvette
    Mar 22, 2023 at 18:13
  • $\begingroup$ @Yvette Thank you for writing an interesting question. That's not easy to do. I agree it's a minor error which doesn't take anything away from the content of the paper. $\endgroup$
    – dipetkov
    Mar 22, 2023 at 18:35
3
$\begingroup$

If including the second variable improves the fit only a tiny amount, yet the degrees of freedom change, that could happen.

I find it easier to take this to the extreme. Add in one variable that really matters and a bunch that do not, say 100. Those 100 irrelevant variables will contribute to the degrees of freedom in the $F$ or $\chi^2$ distribution, making quite high the threshold for model improvement to be significant. However, those 100 irrelevant variables make little difference to the model fit, not enough to make up for the degrees of freedom. Thus, those $100$ irrelevant variables wash out the real effect of the one relevant variable.

For just two variables, the same idea applies, just less dramatically.

EDIT

set.seed(2023)
N <- 20
x1 <- runif(N)
x2 <- runif(N)
x3 <- x2 + runif(N)
x4 <- x2 + runif(N)
x5 <- x2 + runif(N)
x6 <- x2 + runif(N)
x7 <- x2 + runif(N)
x8 <- x2 + runif(N)
x9 <- x2 + runif(N)
y <- x1 + x2 + rnorm(N)
L1 <- lm(y ~ x1)
L2 <- lm(y ~ x1 + x2)
L9 <- lm(y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9)
cor(x2, x3)
anova(L1, L2)$`Pr(>F)`[2] # 0.01619273
anova(L1, L9)$`Pr(>F)`[2] # 0.6096655

In the above simulation, despite the importance of the x2 variable, entering this feature along with unrelated variables leads to the improvement in model fit offered by x2 being washed out by the noise. The p-value for the F-test of adding x2 on its own is a rather significant 0.0162, consistent with how the y variable is simulated to depend on x2, yet the p-value for the F-test of x2 along with unrelated variables is 0.610.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.