4
$\begingroup$

Let’s consider the following regression model:

$y = B_{0} + B_{1}*x$

where

  • $B_{0}$ — represents the intercept
  • $B_{1}$ — represents the coefficient
  • $x$ — represents the independent variable
  • $y$ — represents the output or the dependent variable

or

Multiple linear regression:

enter image description here

Mathematically, R-squared is calculated by dividing the sum of squares of residuals ($SS_{res}$) by the total sum of squares ($SS_{tot}$) and then subtract it from 1. In this case, $SS_{tot}$ measures the total variation. $SS_{res}$ measures explained variation and $SS_{res}$ measures the unexplained variation.

As $SS_{res}+ SS_{res} = SS_{tot}$

$R² = Explained variation / Total Variation$

enter image description here

Adjusted R-Squared can be calculated mathematically in terms of the sum of squares. The only difference between R-square and the Adjusted R-square equation is the degree of freedom.

enter image description here

In the above equation, $df_{t}$ is the degrees of freedom $n– 1$ of the estimate of the population variance of the dependent variable, and $df_{e}$ is the degrees of freedom $n – p – 1$ of the estimate of the underlying population error variance.

Adjusted R-squared value can be calculated based on the value of r-squared, the number of independent variables (predictors), total sample size.

enter image description here

What happens when we introduce more variables to a linear regression model in terms of $R^2$ and adjusted $R^2$?

Will they increase, decrease, or remain constant?

$\endgroup$

3 Answers 3

8
$\begingroup$

If you introduce more variables, the $R^2$ will always increase, it can never decrease. This follows mathematically from the observation that $$ (y-\beta_0-\beta_1 x_1-...-\beta_p x_p-\beta_{p+1} x_{p+1})^2 \leq(y-\beta_0-\beta_1 x_1-...-\beta_p x_p)^2$$

On the other hand, the adjusted $R^2$ makes an adjustement for the number of variables. It will typically increase if your new variable is highly correlated to your response $Y$ and decrease if this new variable is only slightly correlated to your response. Therefore, it is considered as a better measure that standard $R^2$ because the $R^2$ will tend to always increase with the number of new variables.

$\endgroup$
2
  • $\begingroup$ Does this hold also for non-linear models such as Decision Trees, Random Forests etc? Also, the $R^2$ will always increase, assuming you are able to reach the optimal solution. If you optimize your coefficient with Gradient descent (or its variants) it is certainly possible that you might not reach the optimal value each time. $\endgroup$
    – ado sar
    Jul 3 at 18:49
  • $\begingroup$ @adosar: 1. You can certainly use $R^2$ for regression tree. Not Decision Trees, and Random Forest, those are classification methods. 2. "$R^2$ will always increase, assuming you are able to reach the optimal solution" is not correct. Since $R^2$ is originally developed for univariate, you can keep increasing $R^2$ by fitting more noise features. What makes adjusted $R^2$ better is that it avoids overestimation by making an adjustment to the number of features. 3. There is no assumption on the way the model is optimized. $\endgroup$
    – Minh Khôi
    Jul 4 at 4:08
2
$\begingroup$

A very important effect of adding more variables is that you can better adjust for confounding, which means better causal interpretation. Which sometimes might be more important than looking at the $R^2$.

For example, imagine we want a regression of the form $Y = X\beta + \epsilon$ where $X$ and $Y$ is thought of as a random variable. Suppose now that there is some unobserved random variable $Z$ that is related to both $X$ and $Y$ such as in the picture below taken from Wikipedia:

confounder

Then even if $X$ has no direct effect on $Y$, if you don't include $Z$ in the regression you will most likely observe a significant value for $\beta$. After including $Z$, the coefficient of $X$ will be closer to only reflect the effect that is not explained by $Z$.

$\endgroup$
1
  • $\begingroup$ Confounding can be reduced when more confounders are added to the model, which increases the likelihood of identifying true associations among exposures of interest. Causality is difficult to establish and it is safe to say that no single cohort study can establish causality. This should not deter one from creating a causal framework from which a methodical testing process may extend and strongly suggest causality. $\endgroup$
    – Todd D
    Feb 23, 2020 at 18:54
0
$\begingroup$

Adding more variables will increase R Squared whether or not the added variables have any statistically significant effect on the dependent variable. On the other hand, adjusted R Squared can increase or decrease.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.