5
$\begingroup$

Lots of people seem to be asking this. They often seem to get shallow answers that merely assert what is true, instead of drawing or explaining the mechanism. They also seem to not find each other -- if you don't know the answer, it's not obvious that these are all the same question.

Has someone given a high-quality answer that we can refer all these askers to? If not, give it your best shot below.

Here are some related questions. This list is incomplete. Feel free to edit and add to it.

Linear Regression Coefficient changes with additional variables

GAM Interactions : Individual and Combined Interactions are different

Nonsignificant interaction still causes main effect to flip?

Interpreting main effect coefficient in different models

Interactions make terms significant in regression when they should not be

Related but not bang on:

What if interaction wipes out my direct effects in regression?

Understanding Simpson's paradox: Andrew Gelman's example with regressing income on sex and height

$\endgroup$
10
  • 2
    $\begingroup$ Having written a lot about these issues for the last 10+ years, and always choosing to do so by deriving conclusions from definitions and first principles, I wonder why you don't seem to have come across any of those posts. See, e.g., stats.stackexchange.com/a/34813/919, stats.stackexchange.com/a/24529/919, stats.stackexchange.com/a/28493/919, stats.stackexchange.com/a/34523/919, stats.stackexchange.com/a/32237/919, stats.stackexchange.com/a/224958/919, stats.stackexchange.com/a/89766/919, etc. $\endgroup$ – whuber Jan 2 at 22:32
  • $\begingroup$ Thanks -- if you post that as an answer, I will accept it. $\endgroup$ – eric_kernfeld Jan 4 at 12:47
  • $\begingroup$ This is more of a meta question, but do you think it is necessary to do more to help these threads find each other? $\endgroup$ – eric_kernfeld Jan 4 at 12:50
  • 2
    $\begingroup$ I do everything I can in that regard, Eric, by routinely searching for duplicates, supplying links, and so on. Unfortunately the SE system is not set up to motivate or reward such activities, so few undertake them. If you have ideas about how to improve things here on CV, then please post them on Meta; if you have ideas about improving SE sites generally, then consider creating a thread on the SE Meta site. $\endgroup$ – whuber Jan 4 at 13:26
  • 1
    $\begingroup$ I actually understand these topics pretty well, and I was mostly concerned about how to collect people landing on the site with similar issues who don't know how to find the right threads. But, it's pretty clear to me now that I am going about this in the wrong way. I will probably delete the question soon, though I might save all the links somewhere first. $\endgroup$ – eric_kernfeld Jan 5 at 15:39
1
+50
$\begingroup$

Coefficient change

Let some there be some data distributed according to a quadratic curve:

$$y \sim \mathcal{N}(\mu = a+bx+cx^2, \sigma^2 = 10^{-3})$$

For instance with $x \sim \mathcal{U}(0,1)$ and $a=0.2$, $b=0$ and $c=1$. Then a linear curve and a polynomial curve will have very different coefficients for the linear term.

example

set.seed(1)
x <- runif(100, 0,1)
y <- rnorm(100, mean = 0.2+0*x+1*x^2,
                sd = 10^-1.5)
plot(x,y, ylim = c(0,1.5),
     pch = 21, col = 1 , bg = 1, cex = 0.7)

mod1 <- lm(y~x)
mod2 <- lm(y~poly(x,2, raw =TRUE))

xs <- seq(0,10,0.01)
lines(xs,predict(mod1,newdata = list(x = xs)), lty = 2)
lines(xs,predict(mod2,newdata = list(x = xs)),lty =1)

legend(0,1.5,c("y = 0.009 + 1.023 x", "y = 0.193 + 0.016 x + 0.994 x^2"), lty = c(2,1))

Correlation

The reason is that the variables/regressors $x$ and $x^2$ correlate.

The coefficient estimates computed with a linear regression are not a simple correlation (perpendicular projection onto each regressor seperately):

$$\hat{\beta} \neq \alpha = \mathbf{X^t} y$$

(this would give coefficients $\alpha_1$ and $\alpha_2$ in the image below, and these coordinates/coefficients/correlations do not change when you add or remove other regressors)

Using the correlation/projection $\mathbf{X^t}y$ is wrong, because if there is a correlation between the vectors in $\mathbf{X}$, then there will be an overlap between some vectors. This part that overlaps will be redundant and added too much. The predicted value $\hat{y} = \alpha \mathbf{X}$ would be too large.

For this reason there is a correction with a term $(\mathbf{X^t}\mathbf{X})^{-1}$ that accounts for the overlap/correlation between the regressors. This might be clear in the image below which stems from this question: Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

example of overlap

Intuitive view

So the regressors $x$ and $x^2$ both correlate with the data $y$ and they both will be able to express the variation in the dependent data. But when we use them together then we are not gonna add them according to their single independent effects (according to correlation with $y$) because that would be too much.

If we use both $x$ and $x^2$ in the regression then obviously the coefficient for the linear term $x$ should be very small since this is the same in the true relation.

However, when we are not using the quadratic term $x^2$ in the regression (or otherwise add a bias to the coefficient for the quadratic term), then the coefficient for $x$ which correlates somewhat with $x^2$ will partly take correct this (take over) and... the value of the estimate for the coefficient of the linear term will change.

Standard error change (and confidence intervals and p-values)

The errors of the variables may be correlated leading to very large errors in some coefficient when they strongly correlate with others. The matrix $(X^TX)^{−1}$ describes this correlation.

Error in the regression line

The image below shows intuitively how this changes when adding other regressors.

The intercept is the point where a regression line crosses $x=0$.

  • On the left the error of the intercept is the error of the mean of the population.
  • On the right the error of the intercept is the error of the regression line intercept.

change of meaning

Confidence regions for correlated parameters

The next image displays the confidence regions (contrasting with confidence intervals) of the above regression in a 2-D plot. Here it takes into account the correlation between the parameters.

The ellipse shows the confidence region which is a based on a multivariate distribution of the slope and intercept which may be related via a correlation matrix. For illustration an alternative type of region is also show. This is depicted by the box which is based on two single variate distributions assuming independence (now the confidence for the single variables is $\sqrt{0.95}$).

By changing the model from $y = a + bx$ to a shifted model $y = a + b(x-35.5)$ we see that the correlation between the slope and intercept changes. Now the "intercept" coincides with the standard error of the line around the point $x=35.5$ which you see in the image above is smaller.

correlation and confidence regions

#used model and data
set.seed(1)

xt <- seq(0,40,0.1)
x <- c(1:10)+30
y <- 10+0.5*x+rnorm(10,0,3)

See also:

$\endgroup$
1
  • $\begingroup$ I would have closed the question, but then it attracted nice contributions such as this -- a nicely intuitive answer with a helpful collection of related threads! $\endgroup$ – eric_kernfeld Jan 8 at 16:17
3
$\begingroup$

The ordinary least squares solution is simply given by:

$$\beta = (X'X)^{-1}X'y$$

Let's imagine we augment $X_{n\times p}$ with one or more variables $\tilde X_{n\times \tilde p}$, appending its corresponding values as columns, and call the resulting matrix ${X^*}_{n\times p^*}$, $p^* = p + \tilde p$. Now, given enough degrees of freedom, coefficients will be given by:

$$\beta^* = ({X^*}'{X^*})^{-1}{X^*}'y$$

If $\left\{A\right\}_{k,:}$ represents the $k$-th row of a matrix $A$, the $k$-th coefficient in each vector corresponds to:

$$\beta_k = \left\{(X'X)^{-1}\right\}_{k,:}X'y$$

$$\beta^*_k = \left\{({X^*}'{X^*})^{-1}\right\}_{k,:}{X^*}'y$$

Since $X^* \neq X$, then there is no reason for $\beta_k$ and $\beta^*_k$ to be equal.

Notice, however, that it can still be true in a specific case. If the added variables are orthogonal the original independent variables, then $(\beta_k = \beta^*_k | k\leq p)$.

This result can be achieved through:

$${X^*}'{X^*}=\left[ \matrix{ {X}'{X} & X' \tilde X \\ \tilde X' X & \tilde X' \tilde X } \right]$$

Rows from the inverse of this covariance matrix left multiply ${X^*}'$. If $X' \tilde X = \tilde X' X = 0$ the covariance become a block matrix, and it can thus be inverted blockwise:

$$({X^*}'{X^*})^{-1}=\left[ \matrix{ {X}'{X} & \color{red}{0} \\ \color{red}{0} & \tilde X' \tilde X } \right]^{-1}= \left[ \matrix{ ({X}'{X})^{-1} & 0 \\ 0 & (\tilde X' \tilde X) ^{-1} } \right]$$

The full solution becomes:

$$ \begin{align} \beta^* &= ({X^*}'{X^*})^{-1}{X^*}'y=\\ &= \left[ \matrix{ ({X}'{X})^{-1} & 0 \\ 0 & (\tilde X' \tilde X) ^{-1} } \right] \left[\matrix{X'y \\ {\tilde X'y}}\right]=\\ &=\left[\matrix{({X}'{X})^{-1}X'y \\ {(\tilde X' \tilde X) ^{-1}\tilde X'y}}\right]=\left[\matrix{\beta \\ \tilde \beta}\right] \end{align} $$

Thus keeping the identity between both results, as the entries pertaining to $\tilde X$ do not affect the coefficients pertaining to the original $X$.

Since coefficients change, so do CIs and p-values. A more in-depth look into how express things in terms of the hat matrix will lead to it all as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.