79
$\begingroup$

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out.

I have a least squares regression model, with one independent variable and one dependent variable. The relationship is not significant. Now I add a second independent variable. Now the relationship between the first independent variable and the dependent variable becomes significant.

How does this work? This is probably demonstrating some issue with my understanding, but to me, but I do not see how adding this second independent variable can make the first significant.

$\endgroup$
7
  • 5
    $\begingroup$ This is a very widely discussed topic on this site. This is probably due to collinearity. Do a search for "collinearity" and you will find dozens of relevant threads. I suggest reading some of the answers to stats.stackexchange.com/questions/14500/… $\endgroup$
    – Macro
    May 14, 2012 at 18:16
  • 3
    $\begingroup$ possible duplicate of significant predictors become non-significant in multiple logistic regression. There are many threads this is effectively a duplicate of -- that was the closest one I could find in under two minutes $\endgroup$
    – Macro
    May 14, 2012 at 18:20
  • 3
    $\begingroup$ This is sort of the opposite problem of the one in the thread @macro just found, but the reasons are very similar. $\endgroup$
    – Peter Flom
    May 14, 2012 at 18:28
  • 3
    $\begingroup$ @Macro, I think you're right that this may be a duplicate, but I think that the issue here is slightly different from the 2 questions above. The OP doesn't refer to the significance of the model-as-a-whole, nor to variables becoming non-significant w/ additional IV's. I suspect this is not about multicollinearity, but about power or possibly suppression. $\endgroup$ May 14, 2012 at 18:29
  • 3
    $\begingroup$ also, @gung, suppression in a linear models only occurs when there is collinearity - the difference is about interpretation, so "this is not about multicollinearity but about possibly suppression" sets up a misleading dichotomy $\endgroup$
    – Macro
    May 14, 2012 at 18:44

5 Answers 5

96
$\begingroup$

Although collinearity (of predictor variables) is a possible explanation, I would like to suggest it is not an illuminating explanation because we know collinearity is related to "common information" among the predictors, so there is nothing mysterious or counter-intuitive about the side effect of introducing a second correlated predictor into the model.

Let us then consider the case of two predictors that are truly orthogonal: there is absolutely no collinearity among them. A remarkable change in significance can still happen.

Designate the predictor variables $X_1$ and $X_2$ and let $Y$ name the predictand. The regression of $Y$ against $X_1$ will fail to be significant when the variation in $Y$ around its mean is not appreciably reduced when $X_1$ is used as the independent variable. When that variation is strongly associated with a second variable $X_2$, however, the situation changes. Recall that multiple regression of $Y$ against $X_1$ and $X_2$ is equivalent to

  1. Separately regress $Y$ and $X_1$ against $X_2$.

  2. Regress the $Y$ residuals against the $X_1$ residuals.

The residuals from the first step have removed the effect of $X_2$. When $X_2$ is closely correlated with $Y$, this can expose a relatively small amount of variation that had previously been masked. If this variation is associated with $X_1$, we obtain a significant result.


All this might perhaps be clarified with a concrete example. To begin, let's use R to generate two orthogonal independent variables along with some independent random error $\varepsilon$:

n <- 32
set.seed(182)
u <-matrix(rnorm(2*n), ncol=2)
u0 <- cbind(u[,1] - mean(u[,1]), u[,2] - mean(u[,2]))
x <- svd(u0)$u
eps <- rnorm(n)

(The svd step assures the two columns of matrix x (representing $X_1$ and $X_2$) are orthogonal, ruling out collinearity as a possible explanation of any subsequent results.)

Next, create $Y$ as a linear combination of the $X$'s and the error. I have adjusted the coefficients to produce the counter-intuitive behavior:

y <-  x %*% c(0.05, 1) + eps * 0.01

This is a realization of the model $Y \sim_{iid} N(0.05 X_1 + 1.00 X_2, 0.01^2)$ with $n=32$ cases.

Look at the two regressions in question. First, regress $Y$ against $X_1$ only:

> summary(lm(y ~ x[,1]))
...
             Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.002576   0.032423  -0.079    0.937
x[, 1]       0.068950   0.183410   0.376    0.710

The high p-value of 0.710 shows that $X_1$ is completely non-significant.

Next, regress $Y$ against $X_1$ and $X_2$:

> summary(lm(y ~ x))
...
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.002576   0.001678  -1.535    0.136    
x1           0.068950   0.009490   7.265 5.32e-08 ***
x2           1.003276   0.009490 105.718  < 2e-16 ***

Suddenly, in the presence of $X_2$, $X_1$ is strongly significant, as indicated by the near-zero p-values for both variables.

We can visualize this behavior by means of a scatterplot matrix of the variables $X_1$, $X_2$, and $Y$ along with the residuals used in the two-step characterization of multiple regression above. Because $X_1$ and $X_2$ are orthogonal, the $X_1$ residuals will be the same as $X_1$ and therefore need not be redrawn. We will include the residuals of $Y$ against $X_2$ in the scatterplot matrix, giving this figure:

lmy <- lm(y ~ x[,2])
d <- data.frame(X1=x[,1], X2=x[,2], Y=y, RY=residuals(lmy))
plot(d)

Here is a rendering of it (with a little prettification):

SPM

This matrix of graphics has four rows and four columns, which I will count down from the top and from left to right.

Notice:

  • The $(X_1, X_2)$ scatterplot in the second row and first column confirms the orthogonality of these predictors: the least squares line is horizontal and correlation is zero.

  • The $(X_1, Y)$ scatterplot in the third row and first column exhibits the slight but completely insignificant relationship reported by the first regression of $Y$ against $X_1$. (The correlation coefficient, $\rho$, is only $0.07$).

  • The $(X_2, Y)$ scatterplot in the third row and second column shows the strong relationship between $Y$ and the second independent variable. (The correlation coefficient is $0.996$).

  • The fourth row examines the relationships between the residuals of $Y$ (regressed against $X_2$) and other variables:

    • The vertical scale shows that the residuals are (relatively) quite small: we couldn't easily see them in the scatterplot of $Y$ against $X_2$.

    • The residuals are strongly correlated with $X_1$ ($\rho = 0.80$). The regression against $X_2$ has unmasked this previously hidden behavior.

    • By construction, there is no remaining correlation between the residuals and $X_2$.

    • There is little correlation between $Y$ and these residuals ($\rho = 0.09$). This shows how the residuals can behave entirely differently than $Y$ itself. That's how $X_1$ can suddenly be revealed as a significant contributor to the regression.

Finally, it is worth remarking that the two estimates of the $X_1$ coefficient (both equal to $0.06895$, not far from the intended value of $0.05$) agree only because $X_1$ and $X_2$ are orthogonal. Except in designed experiments, it is rare for orthogonality to hold exactly. A departure from orthogonality usually causes the coefficient estimates to change.

$\endgroup$
12
  • $\begingroup$ So, I understand that $X_1$ explains the variability that $X_2$ does not. Does this require that $X_2$ be (more) significant than $X_1$? Or can $X_1$ be made significant without significance of $X_2$? $\endgroup$
    – Ronald
    May 14, 2012 at 23:15
  • 1
    $\begingroup$ @Ronald, I believe the answers to your questions are no and yes, respectively. You can find out by modifying the example in this reply: change the coefficients (0.05 and 0.01) in the model and the number of cases ($n=32$) to see what happens. $\endgroup$
    – whuber
    May 15, 2012 at 11:31
  • 2
    $\begingroup$ @user30609 A great deal has been written about this. A good multiple regression text would include these ideas, such as Tukey & Mosteller's Data Analysis and Regression. Some articles on them have appeared in The American Statistician; a good keyword to use for searching is "Gram-Schmidt." $\endgroup$
    – whuber
    Feb 20, 2020 at 13:48
  • 1
    $\begingroup$ @Confused That's partly correct: the linear algebra machinery doesn't apply directly to logistic regression. But there are a kind of residuals, called deviance residuals, that give information sort of like ordinary least squares residuals. $\endgroup$
    – whuber
    Feb 25, 2021 at 20:57
  • 1
    $\begingroup$ Ok thanks, so it's still possible for a variable to be significant only if it is included alongside another variable in a logistic regression. Thanks! $\endgroup$
    – confused
    Feb 25, 2021 at 22:49
24
$\begingroup$

It feels like the OP's question can be interpreted in two different ways:

  1. Mathematically, how does OLS work, such that adding an independent variable can change results in an unexpected way?

  2. How can modifying my model by adding one variable change the effect of another, independent variable in the model?

There are several good answers already for question #1. And question #2 may be so obvious to the experts that they assume the OP must be asking question #1 instead. But I think question #2 deserves an answer, which would be something like:

Let's start with an example. Say that you had the heights, age, gender, etc, of a number of children, and you wanted to do a regression to predict their height.

You start with a naive model that uses gender as the independent variable. And it's not statistically significant. (How could it be, you're mixing 3-year-olds and teen-agers.)

Then you add in age and suddenly not only is age significant, but so is gender. How could that be?

Of course, in my example, you can clearly see that age is an important factor in the height of a child/teen. Probably the most important factor that you have data on. Gender can matter, too, especially for older children and adults, but gender alone is a poor model of how tall a child is.

Age plus gender is a reasonable (though, of course simplified) model that's adequate for the task. If you add other data -- interaction of age and gender, diet, height of parents, etc -- you could make an even better model, which would of course still be simplified compared to the host of factors that actually determine a child's height, but then again all models are simplified versions of reality. (A map of the world that's 1:1 scale isn't too useful for a traveler.)

Your original model (gender only) is too simplified -- so simplified that it's essentially broken. But that doesn't mean that gender is not useful in a better model.

EDIT: added gung's suggestion re: the interaction term of age and gender.

$\endgroup$
2
  • 1
    $\begingroup$ +1, nb, age & gender will presumably need an interaction term as well. $\endgroup$ Dec 26, 2013 at 16:44
  • 1
    $\begingroup$ +1 That's a great example because it is so simple and intuitively clear, and at the same fits exactly to the situation described in much more detail but only abstractly by @whuber in his accepted answer here. $\endgroup$
    – amoeba
    Jan 10, 2015 at 0:06
24
$\begingroup$

I think this issue has been discussed before on this site fairly thoroughly, if you just knew where to look. So I will probably add a comment later with some links to other questions, or may edit this to provide a fuller explanation if I can't find any.

There are two basic possibilities: First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV. The second possibility is that you have a suppressor variable. This is a very counter-intuitive topic, but you can find some info here*, here or this excellent CV thread.

* Note that you need to read all the way through to the bottom to get to the part that explains suppressor variables, you could just skip ahead to there, but you will be best served by reading the whole thing.


Edit: as promised, I'm adding a fuller explanation of my point regarding how the other IV can absorb some of the residual variability and thus increasing the power of the statistical test of the initial IV. @whuber added an impressive example, but I thought I might add a complimentary example that explains this phenomenon in a different way, which may help some people understand the phenomenon more clearly. In addition, I demonstrate that the second IV does not have to be more strongly associated (although, in practice, it almost always will be for this phenomenon to occur).

Covariates in a regression model can be tested with $t$-tests by dividing the parameter estimate by its standard error, or they can be tested with $F$-tests by partitioning the sums of squares. When type III SS are used, these two testing methods will be equivalent (for more on types of SS and associated tests, it may help to read my answer here: How to interpret type I SS). For those just starting to learn about regression methods, the $t$-tests are often the focus because they seem easier for people to understand. However, this is a case where I think looking at the ANOVA table is more helpful. Let's recall the basic ANOVA table for a simple regression model:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &x_1 &\sum(\hat y_i-\bar y)^2 &1 &\frac{\text{SS}_{x_1}}{\text{df}_{x_1}} &\frac{\text{MS}_{x_1}}{\text{MS}_{\rm res}} \\ &\text{Residual} &\sum(y_i-\hat y_i)^2 &N-(1+1) &\frac{\text{SS}_{\rm res}}{\text{df}_{\rm res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

Here $\bar y$ is the mean of $y$, $y_i$ is the observed value of $y$ for unit (e.g., patient) $i$, $\hat y_i$ is model's predicted value for unit $i$, and $N$ is the total number of units in the study. If you have a multiple regression model with two orthogonal covariates, the ANOVA table might be constructed like so:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &x_1 &\sum(\hat y_{x_{1i}\bar x_2}-\bar y)^2 &1 &\frac{\text{SS}_{x_1}}{\text{df}_{x_1}} &\frac{\text{MS}_{x_1}}{\text{MS}_{\rm res}} \\ &x_2 &\sum(\hat y_{\bar x_1x_{2i}}-\bar y)^2 &1 &\frac{\text{SS}_{x_2}}{\text{df}_{x_2}} &\frac{\text{MS}_{x_2}}{\text{MS}_{\rm res}} \\ &\text{Residual} &\sum(y_i-\hat y_i)^2 &N-(2+1) &\frac{\text{SS}_{\rm res}}{\text{df}_{\rm res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

Here $\hat y_{x_{1i}\bar x_2}$, for example, is the predicted value for unit $i$ if its observed value for $x_1$ was its actual observed value, but its observed value for $x_2$ was the mean of $x_2$. Of course, it is possible that $\bar x_2$ is the observed value of $x_2$ for some observation, in which case there are no adjustments to be made, but this won't typically be the case. Note that this method for creating the ANOVA table is only valid if all variables are orthogonal; this is a highly simplified case created for expository purposes.

If we are considering the situation where the same data are used to fit a model both with and without $x_2$, then the observed $y$ values and $\bar y$ will be the same. Thus, the total SS must be the same in both ANOVA tables. In addition, if $x_1$ and $x_2$ are orthogonal to each other, then $SS_{x_1}$ will be identical in both ANOVA tables as well. So, how is it that there can be sums of squares associated with $x_2$ in the table? Where did they come from if the total SS and $SS_{x_1}$ are the same? The answer is that they came from $SS_\text{res}$. The $\text{df}_{x_2}$ are also taken from $\text{df}_\text{res}$.

Now the $F$-test of $x_1$ is the $MS_{x_1}$ divided by $MS_\text{res}$ in both cases. Since $MS_{x_1}$ is the same, the difference in the significance of this test comes from the change in $MS_\text{res}$, which has changed in two ways: It started with fewer SS, because some were allotted to $x_2$, but those are divided by fewer df, since some degrees of freedom were allotted to $x_2$, as well. The change in the significance / power of the $F$-test (and equivalently the $t$-test, in this case) is due to how those two changes trade off. If more SS are given to $x_2$, relative to the df that are given to $x_2$, then the $MS_\text{res}$ will decrease, causing the $F$ associated with $x_1$ to increase and $p$ to become more significant.

The effect of $x_2$ does not have to be larger than $x_1$ for this to occur, but if it is not, then the shifts in $p$-values will be quite small. The only way it will end up switching between non-significance and significance is if the $p$-values happen to be just slightly on both sides of alpha. Here is an example, coded in R:

x1 = rep(1:3, times=15)
x2 = rep(1:3, each=15)
cor(x1, x2)     # [1] 0
set.seed(11628)
y       = 0 + 0.3*x1 + 0.3*x2 + rnorm(45, mean=0, sd=1)
model1  = lm(y~x1)
model12 = lm(y~x1+x2)

anova(model1)
#  ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  5.314  5.3136  3.9568 0.05307 .
# Residuals 43 57.745  1.3429                  
#  ...
anova(model12)
#  ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  5.314  5.3136  4.2471 0.04555 *
# x2         1  5.198  5.1979  4.1546 0.04785 *
# Residuals 42 52.547  1.2511                  
#  ...

In fact, $x_2$ doesn't have to be significant at all. Consider:

set.seed(1201)
y       = 0 + 0.3*x1 + 0.3*x2 + rnorm(45, mean=0, sd=1)
anova(model1)
# ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  3.631  3.6310  3.8461 0.05636 .
# ...
anova(model12)
# ...
#           Df Sum Sq Mean Sq F value  Pr(>F)  
# x1         1  3.631  3.6310  4.0740 0.04996 *
# x2         1  3.162  3.1620  3.5478 0.06656 .
# ...

These are admittedly nothing like the dramatic example in @whuber's post, but they may help people understand what is going on here.

$\endgroup$
8
  • 1
    $\begingroup$ (+1) for "First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV", which @whuber gave a nice example of $\endgroup$
    – Macro
    May 14, 2012 at 21:41
  • $\begingroup$ (+1) In the beginning you give three links. The first one (external one) is unfortunately broken (404 error). Apart from that: you say that there are "two basic possibilities": the added second IV increases the power to test the first IV (and this is exactly the situation described by Whuber and by Wayne in their answers) or there is a supressor variable (which one, btw? the first or the second?). My question: are these really two distinct situations? Or is it essentially the same thing, maybe viewed a bit differently? It would be great if you could expand on that. $\endgroup$
    – amoeba
    Jan 10, 2015 at 0:25
  • $\begingroup$ @gung, thanks for the reply. Ttnphns starts with giving a link to a paper discussing suppression and some other related effects, and this paper claims that "the most generally accepted definition of a suppressor variable (Tzelgov & Henik, 1991) [is] “a variable which increases the predictive validity of another variable (or set of variables) by its inclusion in a regression equation”". This sounds exactly like what OP here asked about, that's why I was confused by you saying that there might be two different reasons. $\endgroup$
    – amoeba
    Jan 11, 2015 at 0:25
  • 1
    $\begingroup$ @amoeba, the point is that you have 2 different mechanisms. That is, you have 2 different underlying DAGs. The outward manifestation may be similar, & the additional power may be more or less, but the reason why the 2nd variable helps differs b/c the 2nd variable's relationship to x1 & y differs. If it isn't clear, you may need to ask a new question; it is hard to do too much in comments. $\endgroup$ Jan 11, 2015 at 0:45
14
$\begingroup$

This thread has already three excellent answers (+1 to each). My answer is an extended comment and illustration to the point made by @gung (which took me some time to understand):

There are two basic possibilities: First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV. The second possibility is that you have a suppressor variable.

For me, the clearest conceptual way to think about multiple regression is geometric. Consider two IVs $x_1$ and $x_2$, and a DV $y$. Let them be centered, so that we do not need to care about intercept. Then if we have $n$ data points in the dataset, all three variables can be imagined as vectors in $\mathbb R^n$; the length of each vector corresponds to the variance and the angle between any two of them corresponds to the correlation. Crucially, performing multiple OLS regression is nothing else than projecting dependent variable $\mathbf y$ onto the plane spanned by $\mathbf x_1$ and $\mathbf x_2$ (with the "hat matrix" simply being a projector). Readers unfamiliar with this approach can look e.g. in The Elements of Statistical Learning, Section 3.2, or in many other books.

"Enhancement"

The following Figure shows both possibilities listed by @gung. Consider only the blue part at first (i.e. ignore all the red lines):

Enhancement and suppression

Here $\mathbf x_1$ and $\mathbf x_2$ are orthogonal predictors spanning a plane (called "plane $X$"). Dependent variable $\mathbf y$ is projected onto this plane, and its projection OD is what is usually called $\hat y$. Then OD is decomposed into OF (contribution of IV1) and OE (contribution of IV2). Note that OE is much longer than OF.

Now imagine that there is no second predictor $\mathbf x_2$. Regressing $\mathbf y$ onto $\mathbf x_1$ would result in projecting it onto OF as well. But the angle AOC ($\alpha$) is close to $90^\circ$; an appropriate statistical test would conclude that there is almost no association between $y$ and $x_1$ and that $x_1$ is hence not significant.

When $x_2$ is added, the projection OF does not change (because $\mathbf x_1$ and $\mathbf x_2$ are orthogonal). However, to test whether $x_1$ is significant, we now need to look at what is left unexplained after $x_2$. The second predictor $x_2$ explains a large portion of $y$, OE, with only a smaller part EC remaining unexplained. For clarity, I copied this vector to the origin and called it OG: notice that the angle GOF ($\beta$) is much smaller than $\alpha$. It can easily be small enough for the test to conclude that it is "significantly smaller than $90^\circ$", i.e. that $x_1$ is now a significant predictor.

Another way to put it is that the test is now comparing the length of OF to OG, and not to OC as before; OF is tiny and "insignificant" compared to OC, but big enough to be "significant" compared to OG.

This is exactly the situation presented by @whuber, @gung, and @Wayne in their answers. I don't know if this effect has a standard name in the regression literature, so I will call it "enhancement".

Suppression

Notice that in the above, if $\alpha=90^\circ$ then $\beta=90^\circ$ as well; in other words, "enhancement" can only enhance the power to detect significant predictor, but if the effect of $x_1$ alone was exactly zero, it will stay exactly zero.

Not so in suppression.

Imagine that we add $x_3$ to $x_1$ (instead of $x_2$) -- please consider the red part of the drawing. The vector $\mathbf x_3$ lies in the same plane $X$, but is not orthogonal to $\mathbf x_1$ (meaning that $x_3$ is correlated with $x_1$). Since the plane $X$ is the same as before, projection OD of $\mathbf y$ also stays the same. However, the decomposition of OD into contributions of both predictors changes drastically: now OD is decomposed into OF' and OE'.

Notice how OF' is much longer than OF used to be. A statistical test would compare the length of OF' to E'C and conclude that the contribution of $x_1$ is significant. This means that a predictor $x_1$ that has exactly zero correlation with $y$ turns out to be a significant predictor. This situation is (very confusingly, in my opinion!) known as "suppression"; see here as to why: Suppression effect in regression: definition and visual explanation/depiction -- @ttnphns illustrates his great answer with a lot of figures similar to mine here (only better done).

$\endgroup$
5
  • 1
    $\begingroup$ I found your answer to be the easiest to understand with the aid of the geometric interpretation. Superb! $\endgroup$
    – zsljulius
    Jul 27, 2015 at 16:31
  • 1
    $\begingroup$ +1 for helpful illustration. I follow your 'enhancement' section, but not the 'suppression' section. Q1, if $\alpha=0$, wouldn't that mean $y$ lies on the $x_1$ axis, i.e. $y$ is a multiple of $x_1$, and so $cor(x_1,y)=1$? Q2, how is this part an illustration of 'a predictor $x_1$ that has exactly zero correlation with $y$'? $y$ is still correlated with $x_1$ in this section. Have I misunderstood? $\endgroup$
    – qoheleth
    Nov 8, 2016 at 23:40
  • $\begingroup$ @qoheleth Thank you for noticing. It should be $\alpha=90$ in this paragraph. I will edit to fix. $\endgroup$
    – amoeba
    Nov 9, 2016 at 9:32
  • $\begingroup$ I got a bit lost on the sentence "This means that a predictor $x_1$ that has exactly zero correlation with $y$ turns out to be a significant predictor." How does that proceed from the sentence before that and the drawing? How do we know $x_1$ and $y$ are not correlated? $\endgroup$
    – roulette01
    Apr 10, 2021 at 19:28
  • $\begingroup$ After reading your comment, I think you're implying the $\alpha = 90$ extends to the last paragraph as well? And not simply to the brief aside right after the "Suppression" heading? $\endgroup$
    – roulette01
    Apr 10, 2021 at 19:42
1
$\begingroup$

I don't think any of the answers have explicitly mentioned the mathematical intuition for the orthogonal/uncorrelated case, so I will show this here, but I don't believe this answer will be 100% complete.

Suppose that $x_1$ and $x_2$ are uncorrelated, which implies that their centered versions are orthogonal, i.e., $(x_1 - \bar{x}_1 ) \perp (x_2 - \bar{x}_2)$.

Now consider the estimators: $$ \hat{\beta} = (X^TX)^{-1}X^Ty $$

Without loss of generality (constant shifts doesn't affect $\hat{\beta}$), suppose that $X$ here consist of the centered versions of $x_1, x_2$. We can also assume that $X$ here does not include the intercept, which is fine since it's centered and we can simply compute the intercept as $\hat{\beta}_0 = \bar{y}$, so $X \in \mathbb{R}^{n \times 2}$ and $(X^TX)^{-1}$ is diagonal, which will result in the estimators from multiple linear regression being the same as that of separate regression $y$ on $x_1$ and $y$ on $x_2$.

Now consider the t score, which is used to compute p values and measure significance. We have $$ t = \frac{\hat{\beta}_j - \beta_j}{SE(\hat{\beta}_j)} $$

We want to test for $\beta_j \neq 0$, so our null hypothesis is $\beta_j = 0$, and we have

$$ t = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)} $$

As we saw earlier, $\hat{\beta}_j$ doesn't change when a predictor that is orthogonal to $x_j$ is added. So for this situation, the only thing that would affect the t score/significance is $SE(\hat{\beta})$, which we know to be $$ SE(\beta) = \operatorname{var}\left(\hat{\beta} \right) = \sigma^2 (X^TX)^{-1} $$

Again note that $(X^TX)^{-1}$ is a diagonal matrix, so this component remains the same for the simple linear regression and multiple linear regression case, so the only thing that could change the t score is $\sigma^2$. If we know the population variance, then the t score wouldn't change, but we typically estimate the population variance with $$ \hat{\sigma}^2 = \frac{1}{n - p - 1}\sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

The estimator of the population variance is non-decreasing when additional predictors are added (orthogonal or not) -- this is equivalent to that the plain vanilla $R^2$ cannot decrease when additional predictors are added, because the sum of squares of residuals can only remain the same (which occurs when the added predictor can be written as a linear combination of the current predictors) or increase. An intuitive way to think about this is if having $p+1$ predictors with non-zero coefficients would get you a worse fit than a p-subset of these $p+1$ predictors, then least squares would just return the smaller model with p non-zero predictors and 1 zeroed predictor

So we see that $SE(\hat{\beta})$ is non-decreasing, which means that the t score increases monotonically, which would contribute to a decrease in the p-value; however, the degrees of freedom of the t-distribution decreases with increasing predictors, and this results in an increase in p-value. So there are competing effects going on here.

$\endgroup$
1
  • $\begingroup$ +1 Thank you for this interesting observation. $\endgroup$
    – whuber
    May 13, 2021 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.