# Why variance explained solely by a particular independent variable increases when I add more independent variables to the model?

I have a problem understanding what is going on. I am fitting three linear regression models to this dataset.

The models are:

• $M_0: Y \approx a + b_0 X_0$,
• $M_*: Y \approx a + \sum_{i=0}^{3} b_i X_i$,
• $M_+ : Y \approx a + \sum_{i=1}^{3} b_i X_i$,

and fraction of explained variance in each of them is $R^2_0$, $R^2_*$ and $R^2_+$ - respectively.

My intuition is that variance explained solely by $X_0$ in model $M_*$ is gain in explained variance after inclusion of $X_0$ to $M_+$ which is $R^2_* - R^2_+$.

As $X_i$ may be corelated, I would also expect that $X_0$ explains most of the variance when it is the only independent variable in the model: $R^2_0 \ge R^2_* - R^2_+$.

However when I calculate $R^2$ for each model, I get $R^2_0 = 0.15\ldots$ and $R^2_* - R^2_+ = 0.21\ldots$.

Is this possible (given that distribution of independent variables is far from either normal or uniform) or are my calculations wrong?

I use the following Python code for calculations:

import pandas as pd
import sklearn.linear_model as sl

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop('Y', axis=1), DF['Y'])
R2_All = model.score(DF.drop('Y', axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop(['Y', 'X0'], axis=1), DF['Y'])
R2_AllButX0 = model.score(DF.drop(['Y', 'X0'], axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF[['X0']], DF['Y'])
R2_X0 = model.score(DF[['X0']], DF['Y'])

print(R2_X0, R2_All - R2_AllButX0)


[EDIT]

An explaination why I think $R^2_* - R^2_+$ may be a valid indicator of variance explained by $X_0$.

I have been experimenting with partial regression plot.

If we consider $RES_+$ residuals of $M_+$ and $RES_{X_0}$ residuals of model $X_0 \approx a + \sum_{i = 1}{3} b_i X_i$, then we have $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$, where $R^2_{X_0}$ is $\frac{SSR}{SST}$ of model $RES_{X_0} ~ b RES_+$.

Actually, the difference between $R^2_* - R^2_+$ and $(1 - R_+) R^2_{X_0}$ calculated is not 0 but -1.4e-16 - while float point number epsilon is 2.2e-16.

As plot of $RES_+$ vs $RES_{X_0}$ is a partial regression plot, I consider $R^2_{X_0}$ to the fraction of variability not explained by $M_+$ but explained after considering $X_0$. Thus $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$ is the fraction of the total variability additionaly explained by extending of $M_+$ to $M_*$. AFAIK partial plot is supposed to reflect dependence of $Y$ on $X_0$ after removal of influence of $X_{1,2,3}$.

Of course I may be completely wrong here.

[EDIT2]

I have found many sources (like https://onlinecourses.science.psu.edu/stat501/node/296 or lecture notes) which define extra sum of squares $SSR(X_0|X_1)$ as $$SSE(X1) - SSE(X_0, X_1) = SSR(X_0, X_1) - SSR(X1)$$ and use it to quantify variability explained by inclusion of variable $X_0$ to a model containing $X_1$ only.

Since $R^2_i = \frac{SSR_i}{SST}$ and $SST$ is same for all models, it makes perfect sense to treat difference of $R^2$ to quantify the fraction of explained variability.

• Once again, there is absolutely no meaning to arithmetics of correlations. See stats.stackexchange.com/questions/105134/… – Spätzle Jan 25 '18 at 9:58
• @Spätzle: Sources I have found contradict that claim. Please see the second edit. – abukaj Jan 25 '18 at 11:07
• @Spätzle I have browsed the question linked. Note that I am subtracting $R^2$ for nested models. – abukaj Jan 25 '18 at 11:19

$R^2$ isn't a measure of marginal contribution to the model, but rather the squared value of the correlation between the fitted values and the original values ($\rho_{y,\hat{y}}$). You cannot subtract $R^2$ values as it has absolutely no meaning. If you want to asses your assumption, the proper method is F-testing, which (under the assumptions of homoscedastic and normal errors) compares two models' amount of unexplained variance and decides according to the proper F-distribution.
$$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}$$ where $SSE^0$ is the amount of unexplained variance in the partial model, $SSE$ is the amount of unexplained variance in the full model, $r$ is the number of differing variables and $n-p$ is the full model's degrees of freedom.
Now, $R^2=\frac{SSR}{SST}=1-\frac{SSE}{SST}$ so we get $\frac{SSE}{SST}=1-R^2$ and finally $SSE=(1-R^2)SST$. For your models we'll have $SSE^0= (1-R^2_0)SST$ and $SSE= (1-R^2_*)SST$, also $r=3$ and $n-p=58932$: $$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}=\frac{\frac{(1-R^2_0)SST}{r}}{\frac{(1-R^2_*)SST}{n-p}}=\frac{\frac{(1-R^2_0)}{3}}{\frac{(1-R^2_*)}{58932}}$$ get the quantile of the F distribution with (3,58932) DFs to botaion the p-value for your hypothesis $\beta_{1,2,3}=0$.
• I see that you say I am not understanding the meaning of $R^2$ (however I still do not see why they shall not be substracted - please see the edit). Also I am not sure what do you mean by "my assumption" here... I do not want to test whether $\beta_{1,2,3} = 0$ (I can see they are non-zero from t-test corrected for 3 comparisons). Actually I need to measure quantitatively the effect of $X_0$ on $Y$ which can not be explained by the other factors. – abukaj Jan 23 '18 at 18:06