How can adding a 2nd IV make the 1st IV significant?

Question

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out.

I have a least squares regression model, with one independent variable and one dependent variable. The relationship is not significant. Now I add a second independent variable. Now the relationship between the first independent variable and the dependent variable becomes significant.

How does this work? This is probably demonstrating some issue with my understanding, but to me, but I do not see how adding this second independent variable can make the first significant.

Answer 1

Although collinearity (of predictor variables) is a possible explanation, I would like to suggest it is not an illuminating explanation because we know collinearity is related to "common information" among the predictors, so there is nothing mysterious or counter-intuitive about the side effect of introducing a second correlated predictor into the model.

Let us then consider the case of two predictors that are truly orthogonal: there is absolutely no collinearity among them. A remarkable change in significance can still happen.

Designate the predictor variables $X_1$ and $X_2$ and let $Y$ name the predictand. The regression of $Y$ against $X_1$ will fail to be significant when the variation in $Y$ around its mean is not appreciably reduced when $X_1$ is used as the independent variable. When that variation is strongly associated with a second variable $X_2$, however, the situation changes. Recall that multiple regression of $Y$ against $X_1$ and $X_2$ is equivalent to

1. Separately regress $Y$ and $X_1$ against $X_2$.

2. Regress the $Y$ residuals against the $X_1$ residuals.

The residuals from the first step have removed the effect of $X_2$. When $X_2$ is closely correlated with $Y$, this can expose a relatively small amount of variation that had previously been masked. If this variation is associated with $X_1$, we obtain a significant result.

---

All this might perhaps be clarified with a concrete example. To begin, let's use R to generate two orthogonal independent variables along with some independent random error $\varepsilon$:

n <- 32
set.seed(182)
u <-matrix(rnorm(2*n), ncol=2)
u0 <- cbind(u[,1] - mean(u[,1]), u[,2] - mean(u[,2]))
x <- svd(u0)$u
eps <- rnorm(n)

(The svd step assures the two columns of matrix x (representing $X_1$ and $X_2$) are orthogonal, ruling out collinearity as a possible explanation of any subsequent results.)

Next, create $Y$ as a linear combination of the $X$'s and the error. I have adjusted the coefficients to produce the counter-intuitive behavior:

y <-  x %*% c(0.05, 1) + eps * 0.01

This is a realization of the model $Y \sim_{iid} N(0.05 X_1 + 1.00 X_2, 0.01^2)$ with $n=32$ cases.

Look at the two regressions in question. First, regress $Y$ against $X_1$ only:

> summary(lm(y ~ x[,1]))
...
             Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.002576   0.032423  -0.079    0.937
x[, 1]       0.068950   0.183410   0.376    0.710

The high p-value of 0.710 shows that $X_1$ is completely non-significant.

Next, regress $Y$ against $X_1$ and $X_2$:

> summary(lm(y ~ x))
...
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.002576   0.001678  -1.535    0.136    
x1           0.068950   0.009490   7.265 5.32e-08 ***
x2           1.003276   0.009490 105.718  < 2e-16 ***

Suddenly, in the presence of $X_2$, $X_1$ is strongly significant, as indicated by the near-zero p-values for both variables.

We can visualize this behavior by means of a scatterplot matrix of the variables $X_1$, $X_2$, and $Y$ along with the residuals used in the two-step characterization of multiple regression above. Because $X_1$ and $X_2$ are orthogonal, the $X_1$ residuals will be the same as $X_1$ and therefore need not be redrawn. We will include the residuals of $Y$ against $X_2$ in the scatterplot matrix, giving this figure:

lmy <- lm(y ~ x[,2])
d <- data.frame(X1=x[,1], X2=x[,2], Y=y, RY=residuals(lmy))
plot(d)

Here is a rendering of it (with a little prettification):

This matrix of graphics has four rows and four columns, which I will count down from the top and from left to right.

Notice:

• The $(X_1, X_2)$ scatterplot in the second row and first column confirms the orthogonality of these predictors: the least squares line is horizontal and correlation is zero.

• The $(X_1, Y)$ scatterplot in the third row and first column exhibits the slight but completely insignificant relationship reported by the first regression of $Y$ against $X_1$. (The correlation coefficient, $\rho$, is only $0.07$).

• The $(X_2, Y)$ scatterplot in the third row and second column shows the strong relationship between $Y$ and the second independent variable. (The correlation coefficient is $0.996$).

• The fourth row examines the relationships between the residuals of $Y$ (regressed against $X_2$) and other variables:

  • The vertical scale shows that the residuals are (relatively) quite small: we couldn't easily see them in the scatterplot of $Y$ against $X_2$.

  • The residuals are strongly correlated with $X_1$ ($\rho = 0.80$). The regression against $X_2$ has unmasked this previously hidden behavior.

  • By construction, there is no remaining correlation between the residuals and $X_2$.

  • There is little correlation between $Y$ and these residuals ($\rho = 0.09$). This shows how the residuals can behave entirely differently than $Y$ itself. That's how $X_1$ can suddenly be revealed as a significant contributor to the regression.

Finally, it is worth remarking that the two estimates of the $X_1$ coefficient (both equal to $0.06895$, not far from the intended value of $0.05$) agree only because $X_1$ and $X_2$ are orthogonal. Except in designed experiments, it is rare for orthogonality to hold exactly. A departure from orthogonality usually causes the coefficient estimates to change.

Answer 2

It feels like the OP's question can be interpreted in two different ways:

1. Mathematically, how does OLS work, such that adding an independent variable can change results in an unexpected way?

2. How can modifying my model by adding one variable change the effect of another, independent variable in the model?

There are several good answers already for question #1. And question #2 may be so obvious to the experts that they assume the OP must be asking question #1 instead. But I think question #2 deserves an answer, which would be something like:

Let's start with an example. Say that you had the heights, age, gender, etc, of a number of children, and you wanted to do a regression to predict their height.

You start with a naive model that uses gender as the independent variable. And it's not statistically significant. (How could it be, you're mixing 3-year-olds and teen-agers.)

Then you add in age and suddenly not only is age significant, but so is gender. How could that be?

Of course, in my example, you can clearly see that age is an important factor in the height of a child/teen. Probably the most important factor that you have data on. Gender can matter, too, especially for older children and adults, but gender alone is a poor model of how tall a child is.

Age plus gender is a reasonable (though, of course simplified) model that's adequate for the task. If you add other data -- interaction of age and gender, diet, height of parents, etc -- you could make an even better model, which would of course still be simplified compared to the host of factors that actually determine a child's height, but then again all models are simplified versions of reality. (A map of the world that's 1:1 scale isn't too useful for a traveler.)

Your original model (gender only) is too simplified -- so simplified that it's essentially broken. But that doesn't mean that gender is not useful in a better model.

EDIT: added gung's suggestion re: the interaction term of age and gender.

Answer 3

I think this issue has been discussed before on this site fairly thoroughly, if you just knew where to look. So I will probably add a comment later with some links to other questions, or may edit this to provide a fuller explanation if I can't find any.

There are two basic possibilities: First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV. The second possibility is that you have a suppressor variable. This is a very counter-intuitive topic, but you can find some info here*, here or this excellent CV thread.

_{* Note that you need to read all the way through to the bottom to get to the part that explains suppressor variables, you could just skip ahead to there, but you will be best served by reading the whole thing.}

---

Edit: as promised, I'm adding a fuller explanation of my point regarding how the other IV can absorb some of the residual variability and thus increasing the power of the statistical test of the initial IV. @whuber added an impressive example, but I thought I might add a complimentary example that explains this phenomenon in a different way, which may help some people understand the phenomenon more clearly. In addition, I demonstrate that the second IV does not have to be more strongly associated (although, in practice, it almost always will be for this phenomenon to occur).

Covariates in a regression model can be tested with $t$-tests by dividing the parameter estimate by its standard error, or they can be tested with $F$-tests by partitioning the sums of squares. When type III SS are used, these two testing methods will be equivalent (for more on types of SS and associated tests, it may help to read my answer here: How to interpret type I SS). For those just starting to learn about regression methods, the $t$-tests are often the focus because they seem easier for people to understand. However, this is a case where I think looking at the ANOVA table is more helpful. Let's recall the basic ANOVA table for a simple regression model:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &x_1 &\sum(\hat y_i-\bar y)^2 &1 &\frac{\text{SS}_{x_1}}{\text{df}_{x_1}} &\frac{\text{MS}_{x_1}}{\text{MS}_{\rm res}} \\ &\text{Residual} &\sum(y_i-\hat y_i)^2 &N-(1+1) &\frac{\text{SS}_{\rm res}}{\text{df}_{\rm res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

Here $\bar y$ is the mean of $y$, $y_i$ is the observed value of $y$ for unit (e.g., patient) $i$, $\hat y_i$ is model's predicted value for unit $i$, and $N$ is the total number of units in the study. If you have a multiple regression model with two orthogonal covariates, the ANOVA table might be constructed like so:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &x_1 &\sum(\hat y_{x_{1i}\bar x_2}-\bar y)^2 &1 &\frac{\text{SS}_{x_1}}{\text{df}_{x_1}} &\frac{\text{MS}_{x_1}}{\text{MS}_{\rm res}} \\ &x_2 &\sum(\hat y_{\bar x_1x_{2i}}-\bar y)^2 &1 &\frac{\text{SS}_{x_2}}{\text{df}_{x_2}} &\frac{\text{MS}_{x_2}}{\text{MS}_{\rm res}} \\ &\text{Residual} &\sum(y_i-\hat y_i)^2 &N-(2+1) &\frac{\text{SS}_{\rm res}}{\text{df}_{\rm res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

Here $\hat y_{x_{1i}\bar x_2}$, for example, is the predicted value for unit $i$ if its observed value for $x_1$ was its actual observed value, but its observed value for $x_2$ was the mean of $x_2$. Of course, it is possible that $\bar x_2$ is the observed value of $x_2$ for some observation, in which case there are no adjustments to be made, but this won't typically be the case. Note that this method for creating the ANOVA table is only valid if all variables are orthogonal; this is a highly simplified case created for expository purposes.

If we are considering the situation where the same data are used to fit a model both with and without $x_2$, then the observed $y$ values and $\bar y$ will be the same. Thus, the total SS must be the same in both ANOVA tables. In addition, if $x_1$ and $x_2$ are orthogonal to each other, then $SS_{x_1}$ will be identical in both ANOVA tables as well. So, how is it that there can be sums of squares associated with $x_2$ in the table? Where did they come from if the total SS and $SS_{x_1}$ are the same? The answer is that they came from $SS_\text{res}$. The $\text{df}_{x_2}$ are also taken from $\text{df}_\text{res}$.

Now the $F$-test of $x_1$ is the $MS_{x_1}$ divided by $MS_\text{res}$ in both cases. Since $MS_{x_1}$ is the same, the difference in the significance of this test comes from the change in $MS_\text{res}$, which has changed in two ways: It started with fewer SS, because some were allotted to $x_2$, but those are divided by fewer df, since some degrees of freedom were allotted to $x_2$, as well. The change in the significance / power of the $F$-test (and equivalently the $t$-test, in this case) is due to how those two changes trade off. If more SS are given to $x_2$, relative to the df that are given to $x_2$, then the $MS_\text{res}$ will decrease, causing the $F$ associated with $x_1$ to increase and $p$ to become more significant.

The effect of $x_2$ does not have to be larger than $x_1$ for this to occur, but if it is not, then the shifts in $p$-values will be quite small. The only way it will end up switching between non-significance and significance is if the $p$-values happen to be just slightly on both sides of alpha. Here is an example, coded in R:

x1 = rep(1:3, times=15)
x2 = rep(1:3, each=15)
cor(x1, x2)     # [1] 0
set.seed(11628)
y       = 0 + 0.3*x1 + 0.3*x2 + rnorm(45, mean=0, sd=1)
model1  = lm(y~x1)
model12 = lm(y~x1+x2)

anova(model1)
#  ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  5.314  5.3136  3.9568 0.05307 .
# Residuals 43 57.745  1.3429                  
#  ...
anova(model12)
#  ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  5.314  5.3136  4.2471 0.04555 *
# x2         1  5.198  5.1979  4.1546 0.04785 *
# Residuals 42 52.547  1.2511                  
#  ...

In fact, $x_2$ doesn't have to be significant at all. Consider:

set.seed(1201)
y       = 0 + 0.3*x1 + 0.3*x2 + rnorm(45, mean=0, sd=1)
anova(model1)
# ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  3.631  3.6310  3.8461 0.05636 .
# ...
anova(model12)
# ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  3.631  3.6310  4.0740 0.04996 *
# x2         1  3.162  3.1620  3.5478 0.06656 .
# ...

These are admittedly nothing like the dramatic example in @whuber's post, but they may help people understand what is going on here.

Answer 4

This thread has already three excellent answers (+1 to each). My answer is an extended comment and illustration to the point made by @gung (which took me some time to understand):

There are two basic possibilities: First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV. The second possibility is that you have a suppressor variable.

For me, the clearest conceptual way to think about multiple regression is geometric. Consider two IVs $x_1$ and $x_2$, and a DV $y$. Let them be centered, so that we do not need to care about intercept. Then if we have $n$ data points in the dataset, all three variables can be imagined as vectors in $\mathbb R^n$; the length of each vector corresponds to the variance and the angle between any two of them corresponds to the correlation. Crucially, performing multiple OLS regression is nothing else than projecting dependent variable $\mathbf y$ onto the plane spanned by $\mathbf x_1$ and $\mathbf x_2$ (with the "hat matrix" simply being a projector). Readers unfamiliar with this approach can look e.g. in The Elements of Statistical Learning, Section 3.2, or in many other books.

"Enhancement"

The following Figure shows both possibilities listed by @gung. Consider only the blue part at first (i.e. ignore all the red lines):

Here $\mathbf x_1$ and $\mathbf x_2$ are orthogonal predictors spanning a plane (called "plane $X$"). Dependent variable $\mathbf y$ is projected onto this plane, and its projection OD is what is usually called $\hat y$. Then OD is decomposed into OF (contribution of IV1) and OE (contribution of IV2). Note that OE is much longer than OF.

Now imagine that there is no second predictor $\mathbf x_2$. Regressing $\mathbf y$ onto $\mathbf x_1$ would result in projecting it onto OF as well. But the angle AOC ($\alpha$) is close to $90^\circ$; an appropriate statistical test would conclude that there is almost no association between $y$ and $x_1$ and that $x_1$ is hence not significant.

When $x_2$ is added, the projection OF does not change (because $\mathbf x_1$ and $\mathbf x_2$ are orthogonal). However, to test whether $x_1$ is significant, we now need to look at what is left unexplained after $x_2$. The second predictor $x_2$ explains a large portion of $y$, OE, with only a smaller part EC remaining unexplained. For clarity, I copied this vector to the origin and called it OG: notice that the angle GOF ($\beta$) is much smaller than $\alpha$. It can easily be small enough for the test to conclude that it is "significantly smaller than $90^\circ$", i.e. that $x_1$ is now a significant predictor.

Another way to put it is that the test is now comparing the length of OF to OG, and not to OC as before; OF is tiny and "insignificant" compared to OC, but big enough to be "significant" compared to OG.

This is exactly the situation presented by @whuber, @gung, and @Wayne in their answers. I don't know if this effect has a standard name in the regression literature, so I will call it "enhancement".

Suppression

Notice that in the above, if $\alpha=90^\circ$ then $\beta=90^\circ$ as well; in other words, "enhancement" can only enhance the power to detect significant predictor, but if the effect of $x_1$ alone was exactly zero, it will stay exactly zero.

Not so in suppression.

Imagine that we add $x_3$ to $x_1$ (instead of $x_2$) -- please consider the red part of the drawing. The vector $\mathbf x_3$ lies in the same plane $X$, but is not orthogonal to $\mathbf x_1$ (meaning that $x_3$ is correlated with $x_1$). Since the plane $X$ is the same as before, projection OD of $\mathbf y$ also stays the same. However, the decomposition of OD into contributions of both predictors changes drastically: now OD is decomposed into OF' and OE'.

Notice how OF' is much longer than OF used to be. A statistical test would compare the length of OF' to E'C and conclude that the contribution of $x_1$ is significant. This means that a predictor $x_1$ that has exactly zero correlation with $y$ turns out to be a significant predictor. This situation is (very confusingly, in my opinion!) known as "suppression"; see here as to why: Suppression effect in regression: definition and visual explanation/depiction -- @ttnphns illustrates his great answer with a lot of figures similar to mine here (only better done).

Answer 5

I don't think any of the answers have explicitly mentioned the mathematical intuition for the orthogonal/uncorrelated case, so I will show this here, but I don't believe this answer will be 100% complete.

Suppose that $x_1$ and $x_2$ are uncorrelated, which implies that their centered versions are orthogonal, i.e., $(x_1 - \bar{x}_1 ) \perp (x_2 - \bar{x}_2)$.

Now consider the estimators: $$ \hat{\beta} = (X^TX)^{-1}X^Ty $$

Without loss of generality (constant shifts doesn't affect $\hat{\beta}$), suppose that $X$ here consist of the centered versions of $x_1, x_2$. We can also assume that $X$ here does not include the intercept, which is fine since it's centered and we can simply compute the intercept as $\hat{\beta}_0 = \bar{y}$, so $X \in \mathbb{R}^{n \times 2}$ and $(X^TX)^{-1}$ is diagonal, which will result in the estimators from multiple linear regression being the same as that of separate regression $y$ on $x_1$ and $y$ on $x_2$.

Now consider the t score, which is used to compute p values and measure significance. We have $$ t = \frac{\hat{\beta}_j - \beta_j}{SE(\hat{\beta}_j)} $$

We want to test for $\beta_j \neq 0$, so our null hypothesis is $\beta_j = 0$, and we have

$$ t = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)} $$

As we saw earlier, $\hat{\beta}_j$ doesn't change when a predictor that is orthogonal to $x_j$ is added. So for this situation, the only thing that would affect the t score/significance is $SE(\hat{\beta})$, which we know to be $$ SE(\beta) = \operatorname{var}\left(\hat{\beta} \right) = \sigma^2 (X^TX)^{-1} $$

Again note that $(X^TX)^{-1}$ is a diagonal matrix, so this component remains the same for the simple linear regression and multiple linear regression case, so the only thing that could change the t score is $\sigma^2$. If we know the population variance, then the t score wouldn't change, but we typically estimate the population variance with $$ \hat{\sigma}^2 = \frac{1}{n - p - 1}\sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

The estimator of the population variance is non-decreasing when additional predictors are added (orthogonal or not) -- this is equivalent to that the plain vanilla $R^2$ cannot decrease when additional predictors are added, because the sum of squares of residuals can only remain the same (which occurs when the added predictor can be written as a linear combination of the current predictors) or increase. An intuitive way to think about this is if having $p+1$ predictors with non-zero coefficients would get you a worse fit than a p-subset of these $p+1$ predictors, then least squares would just return the smaller model with p non-zero predictors and 1 zeroed predictor

So we see that $SE(\hat{\beta})$ is non-decreasing, which means that the t score increases monotonically, which would contribute to a decrease in the p-value; however, the degrees of freedom of the t-distribution decreases with increasing predictors, and this results in an increase in p-value. So there are competing effects going on here.