relation between $R^2$ of simple regression and multiple regression

Question

A very basic question concerning the $R^2$ of OLS regressions

1. run OLS regression y ~ x1, we have an $R^2$, say 0.3
2. run OLS regression y ~ x2, we have another $R^2$, say 0.4
3. now we run a regression y ~ x1 + x2, what value can this regression's R squared be?

I think it's clear the $R^2$ for the multiple regression should be no less than 0.4, but is it possible for it to be more than 0.7?

Answer 1

The second regressor can simply make up for what the first did not manage to explain in the dependent variable. Here is a numerical example:

Generate x1 as a standard normal regressor, sample size 20. Without loss of generality, take $y_i=0.5x_{1i}+u_i$, where $u_i$ is $N(0,1)$, too. Now, take the second regressor x2 as simply the difference between the dependent variable and the first regressor.

n <- 20 
x1 <- rnorm(n)

y <- .5*x1 + rnorm(n)

x2 <- y - x1
summary(lm(y~x1))$r.squared
summary(lm(y~x2))$r.squared
summary(lm(y~x1+x2))$r.squared

Answer 2

Coefficient of determination in multiple linear regression: In multiple linear regression the coefficient-of-determination can be written in terms of the pairwise correlations for the variables using the quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}},$$

where $\boldsymbol{r}_{\mathbf{y},\mathbf{x}}$ is the vector of correlations between the response vector and each of the explanatory vectors, and $\boldsymbol{r}_{\mathbf{x},\mathbf{x}}$ is the matrix of correlations between the explanatory vectors (for more on this, see this related question). In the case of a bivariate regression you have:

$$\begin{equation} \begin{aligned} R^2 &= \begin{bmatrix} r_{Y,X_1} \\[6pt] r_{Y,X_2} \\[6pt] \end{bmatrix}^\text{T} \begin{bmatrix} 1 & r_{X_1,X_2} \\[6pt] r_{X_1,X_2} & 1 \\[6pt] \end{bmatrix}^{-1} \begin{bmatrix} r_{Y,X_1} \\[6pt] r_{Y,X_2} \\[6pt] \end{bmatrix} \\[6pt] &= \frac{1}{1-r_{X_1,X_2}^2} \begin{bmatrix} r_{Y,X_1} \\[6pt] r_{Y,X_2} \\[6pt] \end{bmatrix}^\text{T} \begin{bmatrix} 1 & -r_{X_1,X_2} \\[6pt] -r_{X_1,X_2} & 1 \\[6pt] \end{bmatrix} \begin{bmatrix} r_{Y,X_1} \\[6pt] r_{Y,X_2} \\[6pt] \end{bmatrix} \\[6pt] &= \frac{1}{1-r_{X_1,X_2}^2} ( r_{Y,X_1}^2 + r_{Y,X_2}^2 - 2 r_{X_1,X_2} r_{Y,X_1} r_{Y,X_2} ). \end{aligned} \end{equation}$$

You did not specify the directions of the univariate correlations in your question, so without loss of generality, we will denote $D \equiv \text{sgn} (r_{Y,X_1}) \cdot \text{sgn} (r_{Y,X_2}) \in \{ -1, +1 \}$. Substituting your values $r_{Y,X_1}^2 = 0.3$ and $r_{Y,X_2}^2 = 0.4$ yields:

$$R^2 = \frac{0.7 - 2 \sqrt{0.12} \cdot D \cdot r_{X_1,X_2}}{1-r_{X_1,X_2}^2}.$$

It is possible for $R^2 > 0.7$, since it is possible for the combined information from the two variables to be more than the sum of its parts. This interesting phenomenon is called 'enhancement' (see e.g., Lewis and Escobar 1986). You can read more generally about the geometric properties of regression quantities in O'Neill (2019).

Answer 3

Other than the lower bound, which is either 0.3 or 0.4 depending on which variable enters the model first, there is not much you can say. How much $R^2$ rises largely depends on the information that the second variable brings into the model. By information, we mean of course the explained variation in the response.

There is one concept that is critical in that regard and that is the correlation between the predictors. If the correlation is large, the new variable will not only bring nothing to the model but it will also complicate inference for your existing variables, as estimates will become imprecise (multicollinearity). This is the reason we would ideally prefer the new variable to be orthogonal to the others. The chances are slim for this to happen in observational studies, but it can be accomplished in controlled settings, e.g. when you are constructing your own experiment.

But how do you quantify precisely the new information a variable will bring to the model? One widely used measure that takes all these into account is the partial $R^2$. If you are familiar with the ANOVA of the linear model, this is nothing more than the proportional decrease in the Error Sum of Squares that you will accomplish by including this variable into your model. High percentages are desirable while low ones will probably make you think whether this is the right course of action.

So as @cardinal pointed out in the comments, your new coefficient of determination could be as high as 1. It could also be as low as 0.400001. There is no way to tell without additional information.

Answer 4

Two ideas

Idea 1: Geometry of Linear Regression Consider the angle between $\theta_1=<y, x1>$, $\theta_2=<y, x2>$ and the angle between $\theta_3=<y, z>$ where z is the projection of y into affine space span by x1,x2 the projection z should be "at most" y itself and "at least" parallels to x2. Since the angle between line and plane is by definition the minimum angle between line and arbitrage other line from plane

Idea 2: Least Square Regression implemented by Successive Orthogonalization (The Elements of Statistical Learning, 2nd edition, page 54) Let's say in step 1 you orthogonalize x2. Depending on the orthogonalized x1 in step 2, you have R2 range from 0.4 and 1.0