How is that I get a low $R^2$ if I use $X1$ against $Y$ and $X2$ against $Y$, but a $R^2 =1$ if I use both together?

How does the combination blow out the $R^2$?

Could you please show me that calculation?

The data:

Y   12.37   12.66   12.00   11.93   11.06   13.03   13.13   11.44   12.86      10.84    11.20   11.56   10.83   12.63   12.46

X1  2.23    2.57    3.87    3.10    3.39    2.83    3.02    2.14    3.04    3.26    3.39    2.35    2.76    3.90    3.16

X2  9.66    8.94    4.40    6.64    4.91    8.52    8.04    9.05    7.71    5.11    5.05    8.51    6.59    4.90    6.96

In multiple regression, you are trying to model observations, $y$ that are supposed to be noisy versions of a prediction, $X\beta$. That is, we should have: $$y=X\beta+\epsilon$$

In your case, $\epsilon$ is almost zero. That is, both explanatory variables almost perfectly explain your observations. You have: $$\beta = (-4.515413640943276 \text{ (constant offset)}, \\ 3.097007886142981, \\ 1.031859030944453)$$ This gives you a nearly perfect fit.

Observations are not noisy

Regarding multicolinearity: By definition, multicolinearity occurs when your predictor variables, $X1$ and $X2$, are linearly related.

Now, in your case, $X1$ and $X2$ do, in fact, have a high correlation coefficient of about -$0.9$ (I had missed this in a previous edit).

Plots of X2 vs X1

However, this is not the cause of the $R2$ jump that you observe by including both $X1$ and $X2$.

For, let multicolinearity have been present. Then, we could write for a small error $\eta$: $$X2 = a + bX1 + \eta$$ In that case, we would be able to write, for each data point $i$: $$y^{(i)} = \beta_0 + \beta_1 X1^{(i)} + \beta_2 (a + bX1^{(i)} + \eta^{(i)}) + \epsilon^{(i)}$$ $$\Rightarrow y^{(i)} = (\beta_0 + a \beta_2) + X1^{(i)} (\beta_1 + b \beta_2) + \epsilon^{(i)} + \beta_2 \eta^{(i)}$$

Now that we know that $\epsilon^{(i)}$ are nearly zero, then, unless $ \beta_2$ were large, had multicolinearity been an issue we would have seen $R2 = 1$ for even the $Y-X1$ regression. Instead, we get $R2 \approx 0$. This means that while $X1$ and $X2$ together are good predictors, they aren't linearly related enough to be good predictors individually.

Having said that, multicolinearity might play a role in other aspects of your problem, given the $X1-X2$ correlation coefficient of $-0.9$.

| cite | improve this answer | |
  • 1
    $\begingroup$ Thanks but as both variables are highly correlated, why do you say there's no collinearity? Why could each variable have so small predictive power alone but perfectly fit together? $\endgroup$ – Luis Sep 14 '16 at 12:42
  • $\begingroup$ I am having trouble reconciling the 3D plot with the scatterplots. The former appears to show linear relations among all three variables whereas the latter show something entirely different. $\endgroup$ – whuber Sep 14 '16 at 12:59
  • $\begingroup$ @whuber, the plots are right. $\endgroup$ – Luis Sep 14 '16 at 13:35
  • $\begingroup$ "both variables are highly correlated": which two do you have in mind? $X1$ and $X2$ are not correlated. The reason I said there is no multi-colinearity is because of the definition of the term: "Multicollinearity exists when two or more of the predictors in a regression model are moderately or highly correlated." The 3 variables are related by a linear equation but it doesn't qualify as multicolinearity as per the definition. $\endgroup$ – Salmonstrikes Sep 14 '16 at 21:58
  • $\begingroup$ @whuber: the 2D scatters are just projections of the 3D one on each plane of the planes. What I was trying to show in the 3D plot was that all the $(X1, X2, Y)$ scatter points lie in a plane (linearly related), yet pairwise, they are not correlated (esp. the $X2-X1$ plot) $\endgroup$ – Salmonstrikes Sep 14 '16 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.