I have three variables A,B & C. My hypothesis states that both, A and B, have an influence on C. What is a good way to show this statistically? Is it safe to calculate two linear regressions in the following way? First I run the regression of A on C and get a certain R² for this model. In the second step I add B to the model, thus I calculate the regression of A and B on C. I expect the R² to be higher than in the previous model. Can I conclude from this, that both variables influence C, as the explained variance is higher?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Is it safe to calculate two linear regressions in the following way? First I run the regression of A on C and get a certain R² for this model. In the second step I add B to the model, thus I calculate the regression of A and B on C. I expect the R² to be higher than in the previous model. Can I conclude from this, that both variables influence C, as the explained variance is higher?
-> No, because the more complex of two nested model M1 must always have the same or higher R2, as it contains the simpler model M0 as a subset.
What you describe here is the principle of ANOVA. The general idea is to add variables, and perform a test whether the increase in R2 or Likelihood is higher than what one would expect under H0 = M0 is the true model.
Read up on ANOVA (e.g. on CV), and note that for the standard Type I ANOVA in R, results differ depending on the order on which you add the factors (see here).
$\begingroup$
$\endgroup$
You can use an F-test of nested models to show that adding additional parameters better explains the data. The test statistic is a function of the sum of squared errors of the complete and reduced models, which is related to the $R^2$.
If n is the number of observations, k is the number of parameters of the reduced model, and p is the additional parameters in the complete model, the statistic is: $F = \frac{ (SSE_{Reduced} - SSE_{Complete})(n - k - p - 1) }{(SSE_{Complete}) (p)}$ and will be distributed as $F_{p,n-k-p-1}$.
$\begingroup$
$\endgroup$
That alone will not allow you to say A or B influence C, but you would be able to use R-squared adjusted (rather than unadjusted) to see if the addition of B improved the model over just A alone. Influence sounds like a causality type statement, which linear regression won't let you claim-- this is just an association and you need subject matter expertise, ideally joined with randomization, to allow a causal claim.
One user mentioned conducting an F-test for nested models: this is generally the right idea. However, in this case you can just look at the t-statistic for B after it's added to the model with A. For a single coefficient tested, the F-statistic for the 1 coefficient will be equal to the square of the t-statistic for that single coefficient, so this will save you one tiny computational step.
If you're going to go to a "complete" model anyway, you may as well fit C as a function of A and B immediately, conduct a global F test to be sure at least one (non-intercept) coefficient is nonzero, and then examine the t-test for each coefficient. But you can also get partial R-squared estimates or do the above to examine adjusted R squared after adding a second variable.
