-1
$\begingroup$

I have a problem understanding what is going on. I am fitting three linear regression models to this dataset.

The models are:

  • $M_0: Y \approx a + b_0 X_0$,
  • $M_*: Y \approx a + \sum_{i=0}^{3} b_i X_i$,
  • $M_+ : Y \approx a + \sum_{i=1}^{3} b_i X_i$,

and fraction of explained variance in each of them is $R^2_0$, $R^2_*$ and $R^2_+$ - respectively.

My intuition is that variance explained solely by $X_0$ in model $M_*$ is gain in explained variance after inclusion of $X_0$ to $M_+$ which is $R^2_* - R^2_+$.

As $X_i$ may be corelated, I would also expect that $X_0$ explains most of the variance when it is the only independent variable in the model: $R^2_0 \ge R^2_* - R^2_+$.

However when I calculate $R^2$ for each model, I get $R^2_0 = 0.15\ldots$ and $R^2_* - R^2_+ = 0.21\ldots$.

Is this possible (given that distribution of independent variables is far from either normal or uniform) or are my calculations wrong?

I use the following Python code for calculations:

import pandas as pd
import sklearn.linear_model as sl

DF = pd.read_csv('data.csv')

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop('Y', axis=1), DF['Y'])
R2_All = model.score(DF.drop('Y', axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF.drop(['Y', 'X0'], axis=1), DF['Y'])
R2_AllButX0 = model.score(DF.drop(['Y', 'X0'], axis=1), DF['Y'])

model = sl.LinearRegression(fit_intercept=True)
model.fit(DF[['X0']], DF['Y'])
R2_X0 = model.score(DF[['X0']], DF['Y'])

print(R2_X0, R2_All - R2_AllButX0)

[EDIT]

An explaination why I think $R^2_* - R^2_+$ may be a valid indicator of variance explained by $X_0$.

I have been experimenting with partial regression plot.

If we consider $RES_+$ residuals of $M_+$ and $RES_{X_0}$ residuals of model $X_0 \approx a + \sum_{i = 1}{3} b_i X_i$, then we have $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$, where $R^2_{X_0}$ is $\frac{SSR}{SST}$ of model $RES_{X_0} ~ b RES_+$.

Actually, the difference between $R^2_* - R^2_+$ and $(1 - R_+) R^2_{X_0}$ calculated is not 0 but -1.4e-16 - while float point number epsilon is 2.2e-16.

As plot of $RES_+$ vs $RES_{X_0}$ is a partial regression plot, I consider $R^2_{X_0}$ to the fraction of variability not explained by $M_+$ but explained after considering $X_0$. Thus $R^2_* - R^2_+ = (1 - R_+) R^2_{X_0}$ is the fraction of the total variability additionaly explained by extending of $M_+$ to $M_*$. AFAIK partial plot is supposed to reflect dependence of $Y$ on $X_0$ after removal of influence of $X_{1,2,3}$.

Of course I may be completely wrong here.

[EDIT2]

I have found many sources (like https://onlinecourses.science.psu.edu/stat501/node/296 or lecture notes) which define extra sum of squares $SSR(X_0|X_1)$ as $$SSE(X1) - SSE(X_0, X_1) = SSR(X_0, X_1) - SSR(X1)$$ and use it to quantify variability explained by inclusion of variable $X_0$ to a model containing $X_1$ only.

Since $R^2_i = \frac{SSR_i}{SST}$ and $SST$ is same for all models, it makes perfect sense to treat difference of $R^2$ to quantify the fraction of explained variability.

$\endgroup$
3
  • 1
    $\begingroup$ Once again, there is absolutely no meaning to arithmetics of correlations. See stats.stackexchange.com/questions/105134/… $\endgroup$
    – Spätzle
    Commented Jan 25, 2018 at 9:58
  • $\begingroup$ @Spätzle: Sources I have found contradict that claim. Please see the second edit. $\endgroup$
    – abukaj
    Commented Jan 25, 2018 at 11:07
  • $\begingroup$ @Spätzle I have browsed the question linked. Note that I am subtracting $R^2$ for nested models. $\endgroup$
    – abukaj
    Commented Jan 25, 2018 at 11:19

1 Answer 1

4
$\begingroup$

$R^2$ isn't a measure of marginal contribution to the model, but rather the squared value of the correlation between the fitted values and the original values ($\rho_{y,\hat{y}}$). You cannot subtract $R^2$ values as it has absolutely no meaning. If you want to asses your assumption, the proper method is F-testing, which (under the assumptions of homoscedastic and normal errors) compares two models' amount of unexplained variance and decides according to the proper F-distribution.

$$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}$$ where $SSE^0$ is the amount of unexplained variance in the partial model, $SSE$ is the amount of unexplained variance in the full model, $r$ is the number of differing variables and $n-p$ is the full model's degrees of freedom.

Now, $R^2=\frac{SSR}{SST}=1-\frac{SSE}{SST}$ so we get $\frac{SSE}{SST}=1-R^2$ and finally $SSE=(1-R^2)SST$. For your models we'll have $SSE^0= (1-R^2_0)SST$ and $SSE= (1-R^2_*)SST$, also $r=3$ and $n-p=58932$: $$F=\frac{\frac{SSE^0}{r}}{\frac{SSE}{n-p}}=\frac{\frac{(1-R^2_0)SST}{r}}{\frac{(1-R^2_*)SST}{n-p}}=\frac{\frac{(1-R^2_0)}{3}}{\frac{(1-R^2_*)}{58932}}$$ get the quantile of the F distribution with (3,58932) DFs to botaion the p-value for your hypothesis $\beta_{1,2,3}=0$.

$\endgroup$
3
  • 1
    $\begingroup$ As a small side note we should add the this version of F-test is only valid under homoscedasticity $\endgroup$
    – Repmat
    Commented Jan 22, 2018 at 9:56
  • $\begingroup$ true, under homoscedasticity and normality. $\endgroup$
    – Spätzle
    Commented Jan 22, 2018 at 10:06
  • $\begingroup$ I see that you say I am not understanding the meaning of $R^2$ (however I still do not see why they shall not be substracted - please see the edit). Also I am not sure what do you mean by "my assumption" here... I do not want to test whether $\beta_{1,2,3} = 0$ (I can see they are non-zero from t-test corrected for 3 comparisons). Actually I need to measure quantitatively the effect of $X_0$ on $Y$ which can not be explained by the other factors. $\endgroup$
    – abukaj
    Commented Jan 23, 2018 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.