2
$\begingroup$

I have three variables A,B & C. My hypothesis states that both, A and B, have an influence on C. What is a good way to show this statistically? Is it safe to calculate two linear regressions in the following way? First I run the regression of A on C and get a certain R² for this model. In the second step I add B to the model, thus I calculate the regression of A and B on C. I expect the R² to be higher than in the previous model. Can I conclude from this, that both variables influence C, as the explained variance is higher?

$\endgroup$

3 Answers 3

3
$\begingroup$

Is it safe to calculate two linear regressions in the following way? First I run the regression of A on C and get a certain R² for this model. In the second step I add B to the model, thus I calculate the regression of A and B on C. I expect the R² to be higher than in the previous model. Can I conclude from this, that both variables influence C, as the explained variance is higher?

-> No, because the more complex of two nested model M1 must always have the same or higher R2, as it contains the simpler model M0 as a subset.

What you describe here is the principle of ANOVA. The general idea is to add variables, and perform a test whether the increase in R2 or Likelihood is higher than what one would expect under H0 = M0 is the true model.

Read up on ANOVA (e.g. on CV), and note that for the standard Type I ANOVA in R, results differ depending on the order on which you add the factors (see here).

$\endgroup$
2
$\begingroup$

You can use an F-test of nested models to show that adding additional parameters better explains the data. The test statistic is a function of the sum of squared errors of the complete and reduced models, which is related to the $R^2$.

If n is the number of observations, k is the number of parameters of the reduced model, and p is the additional parameters in the complete model, the statistic is: $F = \frac{ (SSE_{Reduced} - SSE_{Complete})(n - k - p - 1) }{(SSE_{Complete}) (p)}$ and will be distributed as $F_{p,n-k-p-1}$.

$\endgroup$
2
$\begingroup$

That alone will not allow you to say A or B influence C, but you would be able to use R-squared adjusted (rather than unadjusted) to see if the addition of B improved the model over just A alone. Influence sounds like a causality type statement, which linear regression won't let you claim-- this is just an association and you need subject matter expertise, ideally joined with randomization, to allow a causal claim.

One user mentioned conducting an F-test for nested models: this is generally the right idea. However, in this case you can just look at the t-statistic for B after it's added to the model with A. For a single coefficient tested, the F-statistic for the 1 coefficient will be equal to the square of the t-statistic for that single coefficient, so this will save you one tiny computational step.

If you're going to go to a "complete" model anyway, you may as well fit C as a function of A and B immediately, conduct a global F test to be sure at least one (non-intercept) coefficient is nonzero, and then examine the t-test for each coefficient. But you can also get partial R-squared estimates or do the above to examine adjusted R squared after adding a second variable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.