1
$\begingroup$

I have two linear regression models predicting the same outcome with the same data, but the predictors are different and the sample sizes slightly vary due to missing values on some predictors but not others. One model has an R^2 of .41 and the other an R^2 of .10. I want to say that the first model explains significantly more variance in the outcome than the second. Is this possible? Is there a package or function in R that I could use to test whether these values are significantly different from each other?

I initially tried anova(model1, model2) but realized that was only appropriate for hierarchical models. I need something that can compare R^2 values from models with different Ns and different predictors.

For clarification:

model1 <- lm(DV ~ a + b, data=data) (933 df)

model2 <- lm(DV ~ w + x + y + z, data=data) (888 df)

The model1 R^2 is .10 and the model2 R^2 is .41. The dataset is the same, but the sample sizes/degrees of freedom are different because of missing values (which I can clean up, if needed). Variables a and b are conceptually different from variables w-z, so I don't want to include them all in the same model.

$\endgroup$
3
  • $\begingroup$ Hi Nicolyn and welcome to CV! What do you mean by 'Ns' ? $\endgroup$
    – utobi
    Commented Mar 2, 2023 at 21:06
  • $\begingroup$ Why are you using two models instead of combining the data? Depending on your goal, it might make more sense to combine the data and run one regression. $\endgroup$
    – Dave
    Commented Mar 2, 2023 at 21:58
  • $\begingroup$ The outcome is identical but the sample sizes are different? I think you will need to provide more details on your project to get useful answers. But potentially an interesting question. $\endgroup$
    – Michael M
    Commented Mar 2, 2023 at 22:02

1 Answer 1

1
$\begingroup$

You could compute the confidence interval of both $R^2$s. With a 95% CI, if they don't overlap, then $p<0.05$. But, if both the samples and the predictors differ, then I don't think this is a very good comparison. It would be better to hold-out participants who have all the variables. Then build the two linear models in the remaining participants and predict your outcome in the held-out participants. At that point you can compare the two pearson's $r$s (or whatever performance measure you want to use), to determine which is better. Since the held-out sample is the same for both then it's a fair comparison.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.