0
$\begingroup$

I have bee wondering why in a multivariate OLS-Regression it is not possible for R² to decrease when increasing the number of explanatory variables. The Point is that for example in the model Y=ß0+ß1x1+ß2x2+u, x1 and x2 are also correlated. Thus when using partialling out I only take the part of x1 that is not correlated or let's say orthogonal to x2. I also only take the part of x2 that is orthogonal to x1 before regressing. But what if the explanatory value is very small of x1 keeping x2 constant and of x2 keeping x1 constant as compared to only including one of the two explanatory variables into the model. To better illustrate my thought let's assume a circle C that represents the variation in Y and a circle A that represents the variation in x1 and a circle B that represents the Variation in x2. All of the three circles overlap. In a univariate Regression with only x1 as explanatory variable R² represents the fraction of the area Where A and C overlap to the area of C. In the Regression with x1 and x2 A and B usually overlap as well. Partialling out would mean to eliminate the area where A, B overlap and thus where A, B and C overlap. Thus, if x1 and x2 are strongly correlated the remaining area where A overlaps with C only and B overlaps with C only could be smaller that the area where A and C overlap in the univariate case. Thus it should be possible that R² decreases when increasing the number of explanatory variables.. But it never does..

My question is why and if there is some technique to divide the area where A,B and C overlap to the explanatory variables, how do they do it (e.g. 50:50?)?

$\endgroup$
1
$\begingroup$

I can't really provide a fitting circle analogy. But to see why $R^2$ can't decrease refer to

$$ R^2 = 1 - (\text{SSR}/\text{SST}) $$

where $\text{SSR}$ is the sum of squared residuals and $\text{SST}$ is the total sum of squares. $R^2$ cannot decrease because the sum of squared residuals never increases when additional regressors are added to the model.

For example, the last digit of one’s social security number has nothing to do with one’s hourly wage, but adding this digit to a wage equation will increase $R^2$ (by a little, at least).

Well the $R^2$ can change, see for instance (in R):

x1 = rnorm(100)
x2 = rnorm(100)
y1 = 1 + x1 - x2 + rnorm(100)

r1 = residuals(lm(y1 ~ x2))
r2 = residuals(lm(x1 ~ x2))
# ols
coef(lm(y1 ~ x1 + x2))
# fwl ols
coef(lm(r1 ~ 0 + r2))

The coefficients are equal, but the $R^2$ is different.

The fact that the coefficients are equal is a strange result (but it is always valid in a OLS/GLS setting, and possible more), you have that guarantee from the frisch waugh lovell (FWL) theorem.

The implication is clear in time series: should you detrend you variables or simply include the trend in the regression? By FWL it does not matter, you always get the same coefficients. Actually the implication of the FWL is larger, since it applies to every variable, not just trends (also it does not matter how that variables is scaled/measured).

$\endgroup$
  • $\begingroup$ Thank you for taking the time to answer to my question. However, that does not resolute my confusion about the stated problem. If we for example regress GDP growth on two similar indicators like the unemployment rate that two different institutions estimate in similar but slightly different ways (e.g. one institution defines a person unemployed that works less than 5 hours a week and another defines a person unemployed that works less than 15 hours a week - this is just an example to have two highly correlated but not perfectly correlated variables). Partialling out would imply that I regress $\endgroup$ – Tom Jul 1 '15 at 18:07
  • $\begingroup$ "[....] the two explanatory variables on one another and thereby lose the variation that both explanatory variables explain mutually. In the next step I take the residuals and regress GDP growth on them. In my opinion in this case it is very likely that SSR increases, because the variation in GDP growth that is explained by both unemployment rates at the same time is not being taken into account anymore. I am sure I must be somehow wrong with that, but I don't understand where my mistake is." - I converted this response to a comment & it got truncated. $\endgroup$ – Scortchi - Reinstate Monica Jul 1 '15 at 19:29
  • $\begingroup$ But how can the coefficients be equal after partialling out? Usually after including another variable that is (usually also) correlated to both the dependent and the explanatory variable the coefficients change. This is to have the ceteris paribus condition and also to not have a biased estimator. Am I wrong with that? $\endgroup$ – Tom Jul 1 '15 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.