4
$\begingroup$

Let’s consider the following regression model:

y = B0 + B1*x

where

  • B0 — represents the intercept
  • B1 — represents the coefficient
  • x — represents the independent variable
  • y — represents the output or the dependent variable

or

Multiple linear regression:

enter image description here

Mathematically, R-squared is calculated by dividing the sum of squares of residuals (SSres) by total sum of squares (SStot) and then subtract it from 1. In this case, SStot measures total variation. SSreg measures explained variation and SSres measures unexplained variation.

As $SSres + SSreg = SStot, R² = Explained variation / Total Variation$

enter image description here

Adjusted R-Squared can be calculated mathematically in terms of the sum of squares. The only difference between R-square and the Adjusted R-square equation is the degree of freedom.

enter image description here

In the above equation, dft is the degrees of freedom n– 1 of the estimate of the population variance of the dependent variable, and dfe is the degrees of freedom n – p – 1 of the estimate of the underlying population error variance.

Adjusted R-squared value can be calculated based on value of r-squared, number of independent variables (predictors), total sample size.

enter image description here

What happens when we introduce more variables to a linear regression model in terms of $R^2$ and adjusted $R^2$?

Will they increase, decrease or remain constant?

$\endgroup$
2
$\begingroup$

If you introduce more variables, the $R^2$ will always increase, it can never decrease. This follows mathematically from the observation that $$ (y-\beta_0-\beta_1 x_1-...-\beta_p x_p-\beta_{p+1} x_{p+1})^2 \leq(y-\beta_0-\beta_1 x_1-...-\beta_p x_p)^2$$

On the other hand, the adjusted $R^2$ makes an adjustement for the number of variables. It will typically increase if your new variable is highly correlated to your response $Y$ and decrease if this new variable is only slightly correlated to your response. Therefore, it is considered as a better measure that standard $R^2$ because the $R^2$ will tend to always increase with the number of new variables.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Adding more variables will increase R Squared whether or not the added variables have any statistically significant effect on the dependent variable. On the other hand, adjusted R Squared can increase or decrease.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

A very important effect of adding more variables is that you can better adjust for confounding, which means better causal interpretation. Which sometimes might be more important than looking at the $R^2$.

For example, imagine we want a regression of the form $Y = X\beta + \epsilon$ where $X$ and $Y$ is thought of as a random variable. Suppose now that there is some unobserved random variable $Z$ that is related to both $X$ and $Y$ such as in the picture below taken from Wikipedia:

confounder

Then even if $X$ has no direct effect on $Y$, if you don't include $Z$ in the regression you will most likely observe a significant value for $\beta$. After including $Z$, the coefficient of $X$ will be closer to only reflect the effect that is not explained by $Z$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Confounding can be reduced when more confounders are added to the model, which increases the likelihood of identifying true associations among exposures of interest. Causality is difficult to establish and it is safe to say that no single cohort study can establish causality. This should not deter one from creating a causal framework from which a methodical testing process may extend and strongly suggest causality. $\endgroup$ – Todd D Feb 23 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.