10
$\begingroup$

In a linear multiple regression equation, if the beta weights reflect the contribution of each individual independent variable over and above the contribution of all the other IVs, where in the regression equation is the variance shared by all the IVs that predicts the DV?

For example, if the Venn diagram displayed below (and taken from CV's 'about' page here: https://stats.stackexchange.com/about) were relabeled to be 3 IVs and 1 DV, where would the area with the asterisk enter into the multiple regression equation?

enter image description here

$\endgroup$
  • 4
    $\begingroup$ I don't see a need for a downvote here. I think this question gets at what is going on in multiple regression at a fundamental level, and offers an opportunity to explain something about MR that would otherwise never be discussed. $\endgroup$ – gung - Reinstate Monica Mar 18 '12 at 3:57
8
$\begingroup$

To understand what that diagram could mean, we have to define some things. Let's say that Venn diagram displays the overlapping (or shared) variance amongst 4 different variables, and that we want to predict the level of $Wiki$ by recourse to our knowledge of $Digg$, $Forum$, and $Blog$. That is, we want to be able to reduce the uncertainty (i.e., variance) in $Wiki$ from the null variance down to the residual variance. How well can that be done? That is the question that a Venn diagram is answering for you.

Each circle represents a set of points, and thereby, an amount of variance. For the most part, we are interested in the variance in $Wiki$, but the figure also displays the variances in the predictors. There are a few things to notice about our figure. First, each variable has the same amount of variance--they are all the same size (although not everyone will use Venn diagrams quite so literally). Also, there is the same amount of overlap, etc., etc. A more important thing to notice is that there is a good deal of overlap amongst the predictor variables. This means that they are correlated. This situation is very common when dealing with secondary (i.e., archival) data, observational research, or real-world prediction scenarios. On the other hand, if this were a designed experiment, it would probably imply poor design or execution. To continue with this example for a little bit longer, we can see that our predictive ability will be moderate; most of the variability in $Wiki$ remains as residual variability after all the variables have been used (eyeballing the diagram, I would guess $R^2\approx.35$). Another thing to note is that, once $Digg$ and $Blog$ have been entered into the model, $Forum$ accounts for none of the variability in $Wiki$.

Now, after having fit a model with multiple predictors, people often want to test those predictors to see if they are related to the response variable (although it's not clear this is as important as people seem to believe it is). Our problem is that to test these predictors, we must partition the Sum of Squares, and since our predictors are correlated, there are SS that could be attributed to more than one predictor. In fact, in the asterisked region, the SS could be attributed to any of the three predictors. This means that there is no unique partition of the SS, and thus no unique test. How this issue is handled depends on the type of SS that the researcher uses and other judgments made by the researcher. Since many software applications return type III SS by default, many people throw away the information contained in the overlapping regions without realizing they have made a judgment call. I explain these issues, the different types of SS, and go into some detail here.

The question, as stated, specifically asks about where all of this shows up in the betas / regression equation. The answer is that it does not. Some information about that is contained in my answer here (although you'll have to read between the lines a little bit).

$\endgroup$
  • $\begingroup$ Hello Gung, Thank you for your posting. It is very interesting and opened my eyes in some areas. However, I am having trouble reading between the lines of the posting you linked to. So, my question remains: In a linear multiple regression equation, if the beta weights reflect the contribution of each individual independent variable over and above the contribution of all the other IVs, where in the regression equation is the variance shared by all the IVs that predicts the DV? $\endgroup$ – Joel W. Mar 18 '12 at 21:01
  • $\begingroup$ Yeah, it's going to be pretty hard to see. The point is there's a fundamental difference between the issue of 1 how to partition the SS for testing, & 2 estimating the betas. 1 regards attributing SS to which predictor; 2 picks optimal values for the betas. The overlap shows up in the former, not in the latter. If you regressed $Wiki$ on $Digg$ & saved residuals, then predicted those resids from $Forum$ & saved resids-2, etc. (this is inappropriate, BTW), you would see the betas fluctuating wildly. But Mult Reg estimates all betas simultaneously, so it doesn't show up. $\endgroup$ – gung - Reinstate Monica Mar 18 '12 at 21:46
  • $\begingroup$ If "The overlap shows up in the former not in the latter" how can the regression equation be reflecting the shared variance? If the Betas indicate the contribution of each IV when the effects of all other IVs are statistically removed, what part of the regression formula reflects the predictive power of the removed shared variance? Or, how can the regression equation show what would happen to the predicted Y if you increment one of the IVs by 1 if the overlap is not reflected in the Betas? A third question: in a MR analysis of the data underlying the Venn diagram would the Forum beta = 0? $\endgroup$ – Joel W. Mar 18 '12 at 23:29
  • $\begingroup$ The overlap is in the test, not the betas--I'm not sure how else to put that. Each beta denotes the effect on the response variable of a 1-unit change in the covariate, with everything else held constant; a given beta would almost certainly not be the same if the other covariates were removed from the model. If the Venn diagram accurately reflects the true data generating process, then the true value for $\beta_{F}=0$, but empirical estimates basically never exactly equal 0 in practice. $\endgroup$ – gung - Reinstate Monica Mar 19 '12 at 3:40
  • 1
    $\begingroup$ @MarkWhite, student's answer is mostly OK. The statement that when X1 & X2 are perfectly correlated their betas are half is not correct; when r=1 the model is unidentifiable (cf, here). As r gets closer to 1, the estimated betas will depend on the relationships in the sample data & can vary widely from sample to sample. $\endgroup$ – gung - Reinstate Monica Jun 24 '17 at 12:18
5
$\begingroup$

Peter Kennedy has a nice description of Ballentine/Venn diagrams for regression in his book and JSE article, including cases where they can lead you astray.

The gist is that the starred area variation is thrown away only for estimating and testing the slope coefficients. That variation is added back in for the purpose of predicting and calculating $R^2$.

$\endgroup$
  • $\begingroup$ +1, I would add "for estimating" and testing "the slope coefficients", but the point about it being included in the $R^2$ is a good one. $\endgroup$ – gung - Reinstate Monica Mar 3 '14 at 22:37
  • $\begingroup$ Indeed and done. $\endgroup$ – Dimitriy V. Masterov Mar 3 '14 at 22:45
  • $\begingroup$ Is the starred area used to calculate predicted y? If so, where in the prediction formula does the starred area contribute to the predicted y? Stated differently, what term or terms in the prediction formula reflect the starred area? $\endgroup$ – Joel W. May 24 '17 at 19:22
3
$\begingroup$

I realize this is a (very) dated thread, but since one of my colleagues asked me this very same question this week and finding nothing on the Web that I could point him to, I thought I would add my two cents "for posterity" here. I am not convinced that the answers provided to date answer the OP's question.

I am going to simplify the problem to involve only two independent variables; it is very straight-forward to extend it to more than two. Consider the following scenario: two independent variables (X1 and X2), a dependent variable (Y), 1000 observations, the two independent variables are highly correlated with each other (r=.99), and each independent variable is correlated with the dependent variable (r=.60). Without loss of generality, standardize all variables to a mean of zero and a standard deviation of one, so the intercept term will be zero in each of the regressions.

Running a simple linear regression of Y on X1 will produce an r-squared of .36 and a b1 value of 0.6. Similarly, running a simple linear regression of Y on X2 will produce an r-squared of .36 and a b1 value of 0.6.

Running a multiple regression of Y on X1 and X2 will produce an r-squared of just a wee bit higher than .36, and both b1 and b2 take on the value of 0.3. Thus, the shared variation in Y is captured in BOTH b1 and b2 (equally).

I think the OP may have made a false (but totally understandable) assumption: namely, that as X1 and X2 come closer and closer to being perfectly correlated, their b-values in the multiple regression equation come closer and closer to ZERO. That is not the case. In fact, when X1 and X2 come closer and closer to being perfectly correlated, their b-values in the multiple regression come closer and closer to HALF of the b-value in the simple linear regression of either one of them. However, as X1 and X2 come closer and closer to being perfectly correlated, the STANDARD ERROR of b1 and b2 moves closer and closer to infinity, so the t-values converge on zero. So, the t-values will converge on zero (i.e., no UNIQUE linear relationship between either X1 and Y or X2 and Y), but the b-values converge to half the value of the b-values in the simple linear regression.

So, the answer to the OP's question is that, as the correlation between X1 and X2 approaches unity, EACH of the partial slope coefficients approaches contributing equally to the prediction of the Y value, even though neither independent variable offers any UNIQUE explanation of the dependent variable.

If you wish to check this empirically, generate a fabricated dataset (...I used a SAS macro named Corr2Data.sas ...) which has the characteristics described above. Check out the b values, the standard errors, and the t-values: you will find that they are exactly as described here.

HTH // Phil

$\endgroup$
  • 1
    $\begingroup$ This is a fantastic explanation, thank you. I tried to simulate different situations in R, and I got to the conclusion that you can't get rid of the shared variability if n is too large, or if the correlation between the output (Y) and the shared componenet (X1 and X2) is too high. But why would the t-values reflect anything which is not the unique contributions of X1 and X2, to begin with? If regression t-values reflect the unique contributions of the predictors, we shouldn't see the shared variabiltiy affects t-values at all, but we do. Why is that? $\endgroup$ – Galit Jun 16 '19 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.