9
$\begingroup$

I am trying to interpret the results of an article, where they applied multiple regression to predict various outcomes. However the $\beta$'s (standardized B coefficients defined as $\beta_{x_1} = B_{x_1} \cdot \frac{\mathrm{SD}_{x_1}}{\mathrm{SD}_y}$ where $y$ is the dependent variable and $x_1$ is a predictor) reported don't seem to match the reported $R^2$:

enter image description here

Despite $\beta$ 's of -0.83, -0.29, -0.16, -0.43, 0.25, and -0.29, the reported $R^2$ is only 0.20.

Also, the three predictors: weight, BMI and fat % are multi-collinear, correlated around r=0.8-0.9 with each other within sexes.

Is the $R^2$ value plausible with these $\beta$ 's, or is there no straight relationship between the $\beta$'s and the $R^2$?

Additionally, might problems with the multicollinear predictors affect the $\beta$ of a fourth predictor (VO2max), which is correlated around r=0.4 with the aforementioned three variables?

$\endgroup$
  • $\begingroup$ What is $\beta$ in this context? A beta coefficient (standardised regression)? Or something else? If so then, them you can't really say anything all you get is an interpretation in terms of standard deviations. The fact that the coefficient implies large effects, does not imply a high $R^2$ value $\endgroup$ – Repmat Jul 14 '16 at 17:52
  • 1
    $\begingroup$ ß stands for standardized b coefficients. For a 1 predictor case ß is equal to pearson's r, which is directly relate to R-squared, however in this multivariate case, why don't high ß's imply a high R-squared? $\endgroup$ – Sakari Jukarainen Jul 14 '16 at 18:03
  • 2
    $\begingroup$ No, in one regressor case $\beta$ is not equal to Pearson's correlation: $\beta=\frac{\text{Cov}(y,x)}{\text{Var}(x)}\neq\frac{\text{Cov}(y,x)}{ \sqrt{ \text{Var}(y)\times\text{Var}(x) } }=\rho(y,x)$. The relationship between $\beta$s and $R^2$ is not as simple. $\endgroup$ – Richard Hardy Jul 14 '16 at 19:50
  • 5
    $\begingroup$ @RichardHardy I suspect that the confusion is that Sakari defined $\beta$ to be the standardized regression coefficient. In a bivariate linear regression the regression coefficient ($b$ in Sakari's notation) is $r_{xy}\frac{s_y}{s_x}$, where $r$ is the correlation and $s$ the standard deviation. To standardize a regression coeffcient we divide the coefficient with the standard deviation of $y$ and multiply with that standard deviation of $x$, so only the correlation is left. So Sakari is right. $\endgroup$ – Maarten Buis Jul 15 '16 at 7:29
  • $\begingroup$ I still dont see why you consider this to be wrong? If there is some summary statistics in the paper, you can simply check if the numbers add up. You even provided the formula to do so. You cannot conclude, simply because the effects are large in abosulte terms, that the models does a good job at explaning the variance in y. $\endgroup$ – Repmat Jul 16 '16 at 18:51
15
$\begingroup$

The geometric interpretation of ordinary least squares regression provides the requisite insight.

Most of what we need to know can be seen in the case of two regressors $x_1$ and $x_2$ with response $y$. The standardized coefficients, or "betas," arise when all three vectors are standardized to a common length (which we may take to be unity). Thus, $x_1$ and $x_2$ are unit vectors in a plane $E^2$--they are located on the unit circle--and $y$ is a unit vector in a three dimensional Euclidean space $E^3$ containing that plane. The fitted value $\hat y$ is the orthogonal (perpendicular) projection of $y$ onto $E^2$. Because $R^2$ simply is the squared length of $\hat y$, we needn't even visualize all three dimensions: all the information we need can be drawn in that plane.

Orthogonal regressors

The nicest situation is when the regressors are orthogonal, as in the first figure.

Figure 1, showing the regressors and $\hat y$ as vectors in a plane.

In this and the rest of the figures I will consistently draw the unit disk in white and the regressors as black arrows. $x_1$ will always point directly to the right. The thick red arrows depict the components of $\hat y$ in the $x_1$ and $x_2$ directions: that is, $\beta_1 x_1$ and $\beta_2 x_2$. The length of $\hat y$ is the radius of the gray circle on which it lies--but remember that $R^2$ is the square of that length.

The Pythagorean Theorem asserts

$$R^2 = |\hat y|^2 = |\beta_1 x_1|^2 + |\beta_2 x_2|^2 = \beta_1^2(1)+\beta_2^2(1) = \beta_1^2 + \beta_2^2.$$

Because the Pythagorean Theorem holds in any number of dimensions, this reasoning generalizes to any number of regressors, yielding our first result:

When the regressors are orthogonal, $R^2$ equals the sum of the squares of the betas.

An immediate corollary is that when there is just one regressor--univariate regression--$R^2$ is the square of the standardized slope.

Correlated

Negatively correlated regressors meet at angles greater than a right angle.

Figure 2, showing negatively correlated regressors

It is visually apparent in this image that the sum of squares of the betas is strictly greater than $R^2$. This can be proven algebraically using the Law of Cosines or by working with matrix solution of the Normal Equations.

By making the two regressors almost parallel, we can position $\hat y$ near the origin (for an $R^2$ near $0$) while it continues to have large components in the $x_1$ and $x_2$ direction. Thus, there is no limit to how small $R^2$ might be.

Figure

Let's memorialize this obvious result, our second generality:

When regressors are correlated, $R^2$ may be arbitrarily smaller than the sum of squares of the betas.

However, this is not a universal relation, as the next figure demonstrates.

Figure 3, showing negatively correlated regressors but betas have opposite signs.

Now $R^2$ strictly exceeds the sum of squares of the betas. By drawing the two regressors close together and keeping $\hat y$ between them, we may make the betas both approach $1/2$, even when $R^2$ is close to $1$. Further analysis may require some algebra: I take that up below.

I leave it to your imagination to construct similar examples with positively correlated regressors, which thereby meet at acute angles.

Notice that these conclusions are incomplete: there are limits to how much less $R^2$ may be compared to the sum of squares of the betas. In particular, by examining the possibilities carefully, you may conclude (for a regression with two regressors) that

When the regressors are positively correlated and the betas have a common sign, or when the regressors are negatively correlated and the betas have different signs, $R^2$ must be at least as large as the sum of the squares of the betas.


Algebraic results

Generally, let the regressors be (column vectors) $x_1, x_2, \ldots, x_p$ and the response be $y$. Standardization means (a) each is orthogonal to the vector $(1,1,\ldots,1)^\prime$ and (b) they have unit lengths:

$$|x_i|^2 = |y|^2 = 1.$$

Assemble the column vectors $x_i$ into an $n\times p$ matrix $X$. The rules of matrix multiplication imply that

$$\Sigma = X^\prime X$$

is the correlation matrix of the $x_i$. The betas are given by the Normal Equations,

$$\beta = (X^\prime X)^{-1} X^\prime y = \Sigma^{-1} (X^\prime y).$$

Moreover, by definition, the fit is

$$\hat y = X \beta = X (\Sigma ^{-1} X^\prime y).$$

Its squared length gives $R^2$ by definition:

$$R^2 = |\hat y|^2 = \hat y^\prime \hat y = (X\beta)^\prime (X\beta) = \beta^\prime (X^\prime X)\beta = \beta^\prime \Sigma\beta. $$

The geometric analysis suggested we look for inequalities relating $R^2$ and the sum of squares of the betas,

$$\sum_{i=1}^p \beta_i^2 = \beta^\prime \beta.$$

The $L_2$ norm of any matrix $A$ is given by the sum of squares of its coefficients (basically treating the matrix as a vector of $p^2$ components in a Euclidean space),

$$|A|_2^2 = \sum_{i,j} a_{ij}^2 = \operatorname{tr}(A^\prime A) = \operatorname{tr}(AA^\prime).$$

The Cauchy-Schwarz Inequality implies

$$R^2 = \operatorname{tr}(R^2) = \operatorname{tr}(\beta^\prime \Sigma \beta) = \operatorname{tr}(\Sigma \beta \beta^\prime) \le |\Sigma|_2 | \beta\beta^\prime|_2 = |\Sigma|_2 \beta^\prime \beta.$$

Since squared correlation coefficients cannot exceed $1$ and there are just $p^2$ of them in the $p\times p$ matrix $\Sigma$, $|\Sigma|_2$ cannot exceed $\sqrt{1\times p^2} = p$. Therefore

$$R^2 \le p\, \beta^\prime \beta.$$

The inequality is attained, for instance, when all the $x_i$ are perfectly positively correlated.

There is an upper limit on how large $R^2$ may be. Its average value per regressor, $R^2/p$, cannot exceed the sum of squares of the standardized coefficients.


Conclusions

What may we conclude in general? Evidently, information about the correlation structure of the regressors as well as the signs of the betas could be used either to bound the possible values of $R^2$ or even to compute it exactly. Absent that full information, little can be said beyond the obvious fact that when the regressors are linearly independent, a single nonzero beta implies $\hat y$ is nonzero, demonstrating $R^2$ is nonzero.

One thing we can definitely conclude from the output in the question is that the data are correlated: because the sum of squares of the betas, equal to $1.1301$, exceeds the maximum possible value of $R^2$ (namely $1$), there must be some correlation.

Another thing is that since the largest beta (in size) is $-0.83$, whose square is $0.69$--far exceeding the reported $R^2$ of $0.20$--we may conclude that some of the regressors must be negatively correlated. (In fact, $\text{VO}_{2\,\text{max}}$ is likely strongly negatively correlated with age, weight, and fat in any sample that covers a wide range of values of the latter.)

If there were only two regressors, we could deduce a great deal more about $R^2$ from knowledge of high regressor correlations and inspection of the betas, because this would enable us to draw an accurate sketch of how $x_1$, $x_2$, and $\hat y$ must be situated. Unfortunately, the additional regressors in this six-variable problem complicate things considerably. In analyzing any two of the variables, we have to "take out" or "control for" the other four regressors (the "covariates"). In so doing, we shorten all of $x_1$, $x_2$, and $y$ by unknown amounts (depending on how all three of those are related to the covariates), leaving us knowing almost nothing about the actual sizes of the vectors we are working with.

$\endgroup$
  • $\begingroup$ +1 but I don't understand why, in the non-orthogonal case, you project $\hat y$ vector orthogonal to the predictor axes, as opposed to making the projecting dotted lines go parallel to the other predictor. This sounds cumbersome but I think you will see what I mean. Your "projections" (two smaller red vectors) do not some up to get the big red $\hat y$ vector. $\endgroup$ – amoeba Jul 15 '16 at 15:46
  • $\begingroup$ @amoeba You are quite right. I have been too hasty in creating these images! I will (hopefully temporarily) delete this post until I get an opportunity to correct the problem. Thank you for pointing this out. $\endgroup$ – whuber Jul 15 '16 at 15:59
  • $\begingroup$ @Amoeba I have corrected the pictures and modified the analysis to match them. Although the details have changed substantially, the conclusions remain the same. $\endgroup$ – whuber Jul 16 '16 at 18:33
  • 1
    $\begingroup$ @amoeba Again you are correct. At some risk of losing interested readers, but now feeling compelled to quantify the geometric intuition, I have tightened that conclusion and justified it with a little bit of algebra. (I trust the algebra is correct!) $\endgroup$ – whuber Jul 17 '16 at 13:48
  • 1
    $\begingroup$ Thanks a lot! As a sidenote, VO2max is negatively correlated with weight and BMI, since they are associated with higher lean body mass. In the said table VO2max actually corresponds to VO2max divided by weight (which is a bad way of scaling VO2max to body size). The VO2max/weight in the table is negatively correlated with all the other predictors, except sex, which might explain the high ß but low R-squared, as you mentioned. $\endgroup$ – Sakari Jukarainen Jul 20 '16 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.