3
$\begingroup$

Background

In regression analysis, $R^2$, the squared multiple correlation, represents the proportion of explained variance by the regression model. Most software's default setting uses Type-III sums of squares (SS), as has been explained and illustrated excellently in other CV questions (e.g. here and here). This effectively means that 'overlapping' explained variance is discarded: if two predictors both explain the same variance in the criterion, the software cannot know which predictor the explained variance 'belongs to' and it is discarded from the model.

Because of these "Type-III SS" dynamics, each regression coefficient represents the unique contribution of each predictor to the explanation of the variance in the dependent variable.

Using these regression coefficients, it is possible to construct the regression equation:

$$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2$$

In this equation, $\hat{y}$ represents the predicted value of the dependent variable ($y$), $x_1$ and $x_2$ represent two predictors, $\beta_0$ represents the intercept (the predicted value of $y$ if both predictors are $0$), and $\beta_1$ and $\beta_2$ represent the regression coefficients of both predictors.

$\beta_1$ and $\beta_2$ only represent the unique contribution of each predictor to the prediction of $\hat{y}$ - the overlapping variance between these two predictors has been removed.

One can complete this equation for each observation and store the resulting predicted ($\hat{y}$) values. It is then possible to compute the correlation between these $\hat{y}$ values and the observed $y$ values. This correlation is called the multiple correlation, $R$, and it's square, $R^2$ is the proportion explained variance.

This proportion explained variance, $R^2$, represents the variance explained in $y$ by both $x_1$ and $x_2$, i.e. by the full model, *including the overlap in those predictors$.

If the latter were not the case, situations would occur where $R^2$ is lower than the bivariate correlation of one or both predictor(s) with the criterion ($y$). And this never happens (right?).

Now, what puzzles me and a friend is the following.

Question

Since $\hat{y}$ is computed using regression coefficients that only account for unique explained variance, and $r_{y \hat{y}}^2$ (i.e. $R^2$) represents the proportion of explained variance by both predictors including shared explained variance, where does this shared explained variance come from?

If $\beta_1$ and $\beta_2$ only represent the unique contribution of each predictor, how does the shared contribution suddenly end up in $R^2$? Wouldn't this require another term in the regression equation? $\beta_0$ is constant, so by definition cannot explain any variance in $y$. Or did we always misunderstand and does $R^2$ actually not represent the the full yellow and red circled in Gung's example here, but instead only the outer 'half moon' sections?

I realise that this question may be a subtle variation on the questions already answered so excellently and clearly by Gung (see links above), but despite reading these two, having Googled this a few times, and having discussed this with friends, we don't manage to figure this out.

$\endgroup$
  • $\begingroup$ Re "$\beta_1$ and $\beta_2$ only represent the unique contribution of each predictor to the prediction of $\hat y$ - the overlapping variance between these two predictors has been removed." This statement is true only when the $X_i$ are orthogonal (that is, there is no "overlapping variance"). When they are not orthogonal, there is no sense in which the variables have "unique" contributions. $\endgroup$ – whuber Jan 17 '18 at 19:20
  • $\begingroup$ Not sure I understand this. How is this consistent with Gung's explanation "Type III SS only gives A those SS that are uniquely attributable to A" - or isn't it? $\endgroup$ – Matherion Jan 18 '18 at 8:02
  • $\begingroup$ @gung is clear, indeed emphatic, about this issue, but if you don't mind I'll add some emphasis: "[Y]our two factors are correlated with each other. ... The problem with your factors being correlated is that there are sums of squares (SS) that are associated with both A and B. ... [S]ince your factors (still only A and B here) are not orthogonal there is no unique partition of these SS. In fact, there can be very many partitions, and if you are willing to slice your SS up into fractions ..., there are infinite partitions." $\endgroup$ – whuber Jan 18 '18 at 14:33
  • $\begingroup$ Ah, I understand the confusion, sorry - I wasn't clear. In psychology, with 'unique contributions' we mean 'contribution not shared by other variables', so, the crescent-shaped red and yellow parts of the circles. So, a 'unique contribution' refers to the covariance a predictor shares with the criterion but with no other predictor. That does exist with correlated predictors, right (i.e. that 'unique' bit is equal to the total covariance if the predictors are orthogonal, and smaller if they're not)? $\endgroup$ – Matherion Jan 18 '18 at 14:48
  • $\begingroup$ Thank you for explaining. However, in that sense of "unique," the $\beta_i$ do not "represent the unique contribution of each vector," as predicated in your question. You might be thinking of the partial correlations rather than the regression coefficients $\beta_i$. $\endgroup$ – whuber Jan 18 '18 at 15:33
3
$\begingroup$

I hope I got you right: let $X$ be the covariates matrix and $y$ the response variable. The OLS coefficients estimate is defined as $\hat{\beta}=(X^TX)^{-1}X^Ty$ and the predicted values are defined $\hat{y}=X\hat{\beta}=X(X^TX)^{-1}X^Ty$, which is the projection of $y$ to the subspace spanned by $X$ columns. Under the normal model you also get $\hat{\beta}\sim~N(\beta,\sigma^2(X^TX)^{-1})$ and $\hat{y}\sim~N(\mu,\sigma^2X(X^TX)^{-1}X^T)$.

When observing the marginal distributions, we get $\hat{\beta}_j\sim~N(\beta,\sigma^2(X^TX)^{-1}_{jj})$ and $\hat{y}_i\sim~N(\mu_i,\sigma^2x_i(X^TX)^{-1}x_i^T)$, but this does not mean the $\beta$ variances matrix is diagonal (and the same applies for the predictions. In fact, when discussing GLM submodels (linear regression included) it is highly unlikely to encounter diagonal covariance matrices.

Now, let $e$ be the residuals: $e=y-\hat{y}=y-X(X^TX)^{-1}X^Ty=(I-X(X^TX)^{-1}X^T)y$.

$SSE=e^Te=y^T(I-X(X^TX)^{-1}X^T)^T(I-X(X^TX)^{-1}X^T)y= y^T(I^T-(X(X^TX)^{-1}X^T)^T)(I-X(X^TX)^{-1}X^T)y= y^T(I-(X(X^TX)^{-1}X^T))(I-X(X^TX)^{-1}X^T)y= y^T(I-X(X^TX)^{-1}X^T)y$

$R^2$ is defined as $R^2=1-\frac{SSE}{SST}$, where $SST=\sum_i{(y_i-\bar{y})^2}$.

Now to some intuitive handwaving: As you can see, $SSE$ "contains information" from the whole $\beta$ variances matrix (i.e, both unique and shared) and not just the diagonals (which stand for unique contributions). This explains how the shared contribution ends up in $R^2$.

Leaving the algebra aside, let me try and simplify the math: $SSE=\sum_i{(y_i-\hat{y}_i)^2}$, this is the sum of squared prediction error, so actually $R^2=1-\frac{SSE}{SST}$ is computed using the regression equation.

Furthermore, as $X$ is the predictors matrix ($x_1, x_2$, etc.) and the regression coefficients are computed as a bunch using the whole $X$ matrix, then each coefficient contains some covariance information. The situation where each coefficient contains only unique information can occur only if there's no covariance or if you compute separate regression for each coefficient which is very wrong.

$\endgroup$
  • $\begingroup$ My apologies, but I'm afraid this is beyond me (I'm a psychologist, unfortunately with no linear algebra training). What are $X^T$, $I$ and $y^T$? In any case, this explanation is probably very useful to more mathematically versed future visitors :-) Could you perhaps also try to explain this on a conceptual level? In the regression equation, do the regression coefficients contain information about the covariance they share with both the criterion and each other, after all? (I know $R^2$ isn't really computed using the regression equation and $r_{y \hat{y}}$, but it could be, right?) $\endgroup$ – Matherion Jan 18 '18 at 8:10
  • $\begingroup$ See my updated answer. $\endgroup$ – Spätzle Jan 18 '18 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.