I have a problem understanding what is going on. I am fitting three linear regression models to this dataset.

The models are:

  • $M_0: Y \approx a + b_0 X_0$,
  • $M_*: Y \approx a + \sum_{i=0}^{3} b_i X_i$,
  • $M_+ : Y \approx a + \sum_{i=1}^{3} b_i X_i$,

and fraction of explained variance in each of them is $R^2_0$, $R^2_*$ and $R^2_+$ - respectively.

My intuition is that variance explained solely by $X_0$ in model $M_*$ is gain in explained variance after inclusion of $X_0$ to $M_+$ which is $R^2_* - R^2_+$.

As $X_i$ may be corelated, I would also expect that $X_0$ explains most of the variance when it is the only independent variable in the model: $R^2_0 \ge R^2_* - R^2_+$.

However when I calculate $R^2$ for each model, I get $R^2_0 = 0.15\ldots$ and $R^2_* - R^2_+ = 0.21\ldots$.

Is this possible (given that distribution of independent variables is far from either normal or uniform) or are my calculations wrong?

I use the following Python code for calculations:

import pandas as pd
import sklearn.linear_model as sl

DF = pd.read_csv('data.csv')

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop('Y', axis=1), DF['Y'])
R2_All = model.score(DF.drop('Y', axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop(['Y', 'X0'], axis=1), DF['Y'])
R2_AllButX0 = model.score(DF.drop(['Y', 'X0'], axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF[['X0']], DF['Y'])
R2_X0 = model.score(DF[['X0']], DF['Y'])

print(R2_X0, R2_All - R2_AllButX0)


An explaination why I think $R^2_* - R^2_+$ may be a valid indicator of variance explained by $X_0$.

I have been experimenting with partial regression plot.

If we consider $RES_+$ residuals of $M_+$ and $RES_{X_0}$ residuals of model $X_0 \approx a + \sum_{i = 1}{3} b_i X_i$, then we have $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$, where $R^2_{X_0}$ is $\frac{SSR}{SST}$ of model $RES_{X_0} ~ b RES_+$.

Actually, the difference between $R^2_* - R^2_+$ and $(1 - R_+) R^2_{X_0}$ calculated is not 0 but -1.4e-16 - while float point number epsilon is 2.2e-16.

As plot of $RES_+$ vs $RES_{X_0}$ is a partial regression plot, I consider $R^2_{X_0}$ to the fraction of variability not explained by $M_+$ but explained after considering $X_0$. Thus $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$ is the fraction of the total variability additionaly explained by extending of $M_+$ to $M_*$. AFAIK partial plot is supposed to reflect dependence of $Y$ on $X_0$ after removal of influence of $X_{1,2,3}$.

Of course I may be completely wrong here.


I have found many sources (like https://onlinecourses.science.psu.edu/stat501/node/296 or lecture notes) which define extra sum of squares $SSR(X_0|X_1)$ as $$SSE(X1) - SSE(X_0, X_1) = SSR(X_0, X_1) - SSR(X1)$$ and use it to quantify variability explained by inclusion of variable $X_0$ to a model containing $X_1$ only.

Since $R^2_i = \frac{SSR_i}{SST}$ and $SST$ is same for all models, it makes perfect sense to treat difference of $R^2$ to quantify the fraction of explained variability.

  • $\begingroup$ Once again, there is absolutely no meaning to arithmetics of correlations. See stats.stackexchange.com/questions/105134/… $\endgroup$ – Spätzle Jan 25 '18 at 9:58
  • $\begingroup$ @Spätzle: Sources I have found contradict that claim. Please see the second edit. $\endgroup$ – abukaj Jan 25 '18 at 11:07
  • $\begingroup$ @Spätzle I have browsed the question linked. Note that I am subtracting $R^2$ for nested models. $\endgroup$ – abukaj Jan 25 '18 at 11:19

$R^2$ isn't a measure of marginal contribution to the model, but rather the squared value of the correlation between the fitted values and the original values ($\rho_{y,\hat{y}}$). You cannot subtract $R^2$ values as it has absolutely no meaning. If you want to asses your assumption, the proper method is F-testing, which (under the assumptions of homoscedastic and normal errors) compares two models' amount of unexplained variance and decides according to the proper F-distribution.

$$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}$$ where $SSE^0$ is the amount of unexplained variance in the partial model, $SSE$ is the amount of unexplained variance in the full model, $r$ is the number of differing variables and $n-p$ is the full model's degrees of freedom.

Now, $R^2=\frac{SSR}{SST}=1-\frac{SSE}{SST}$ so we get $\frac{SSE}{SST}=1-R^2$ and finally $SSE=(1-R^2)SST$. For your models we'll have $SSE^0= (1-R^2_0)SST$ and $SSE= (1-R^2_*)SST$, also $r=3$ and $n-p=58932$: $$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}=\frac{\frac{(1-R^2_0)SST}{r}}{\frac{(1-R^2_*)SST}{n-p}}=\frac{\frac{(1-R^2_0)}{3}}{\frac{(1-R^2_*)}{58932}}$$ get the quantile of the F distribution with (3,58932) DFs to botaion the p-value for your hypothesis $\beta_{1,2,3}=0$.

  • 1
    $\begingroup$ As a small side note we should add the this version of F-test is only valid under homoscedasticity $\endgroup$ – Repmat Jan 22 '18 at 9:56
  • $\begingroup$ true, under homoscedasticity and normality. $\endgroup$ – Spätzle Jan 22 '18 at 10:06
  • $\begingroup$ I see that you say I am not understanding the meaning of $R^2$ (however I still do not see why they shall not be substracted - please see the edit). Also I am not sure what do you mean by "my assumption" here... I do not want to test whether $\beta_{1,2,3} = 0$ (I can see they are non-zero from t-test corrected for 3 comparisons). Actually I need to measure quantitatively the effect of $X_0$ on $Y$ which can not be explained by the other factors. $\endgroup$ – abukaj Jan 23 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.