# Adding a correlated variable to the regression?

I am running a regression like below: $Y = \beta_0+\beta_1X_1+\beta_2X_2 +\beta_3X_3+\epsilon$, where $X_1$ is a gender dummy and the result shows it has NO statistic significance.

Then I added one more variable $X_1\cdot X_3$ to the model above (with the purpose to test whether the effect of $X_3$ on Y differs by gender), but was surprised that in the new model, all the variables became statistically significant. I just think that the new variable $X_1X_3$ should have even reduced the significance of the dummy $X_1$ because these two variables have high correlation. So why could it happen?

• stats.stackexchange.com/questions/160026/… this might fit your purposes. Roughly spoken, you are changing your reference category and therefore the interpretation of the dummy variables changes. Jan 30, 2018 at 13:37

Think of something like this:

n <- 500
women <- n/2
X.women <- runif(women)
X.men <- runif(n-women)
beta3.women <- 2
beta3.men <- -beta3.women
beta0 <- 1
beta1 <- -beta0
y.women <- beta1 + beta3.women*X.women + rnorm(women, sd=.1)
y.men <- beta0 + beta3.men*X.men + rnorm(n-women, sd=.1)
y <- c(y.women,y.men)
X <- c(X.women,X.men)

plot(X.women,y.women, col="magenta", pch=19, xlab="X", ylab="y")
points(X.men,y.men, col="blue", pch=19)
women.dummy <- c(rep(1,women),rep(0,n-women))

summary(lm(y~X+women.dummy))
summary(lm(y~women.dummy+X+I(women.dummy*X)))


The slopes and intercepts of this artificial dataset are opposite in sign, so that, on average, when fitting no interaction, the regression line is about flat, with no real need for a different intercept for men and women.

Once we introduce the interaction, separate lines are fit for men and women, with evidently both significant intercepts and slopes. (This ignores $X_2$, which seems tangential to the story.)

> summary(lm(y~X+women.dummy))

Call:
lm(formula = y ~ X + women.dummy)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.07295    0.05595   1.304    0.193
X            0.01416    0.08995   0.157    0.875
women.dummy -0.02515    0.05338  -0.471    0.638

> summary(lm(y~women.dummy+X+I(women.dummy*X)))

Call:
lm(formula = y ~ women.dummy + X + I(women.dummy * X))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)         0.99616    0.01189   83.81   <2e-16 ***
women.dummy        -1.98039    0.01763 -112.34   <2e-16 ***
X                  -1.98613    0.02174  -91.36   <2e-16 ***
I(women.dummy * X)  3.95259    0.03056  129.34   <2e-16 ***


Somewhat more theoretically, the OLS estimator $\hat\beta=(X'X)^{-1}X'y=(X'X/n)^{-1}X'y/n$ will tend to zero (and hence not be likely to produce significant results) if $X'y/n\to_p0$.

In the present context $$\frac{X'y}{n}=\begin{pmatrix}\overline{Y}\\ \overline{Y}_w\\ \overline{X_3y}\end{pmatrix},$$ where $\overline{Y}_w$ denotes the sample average for women and $$\overline{X_3y}=\frac{1}{n}\sum_{i=1}^nX_{3i}y_i$$ Combining the true models for men $$y_i=1-2X_{3i}+u_i$$ and women, $$y_i=-1+2X_{3i}+u_i$$ into a joint model gives, with $W_i$ the female dummy, (so that setting $W_i=1$ retrieves the model for women and setting $W_i=0$ that for men) $$y_i=1-2\cdot W_i-2X_{3i}+4X_{3i}W_i+u_i$$ Thus, $$\frac{1}{n}\sum_{i=1}^nX_{3i}y_i=\frac{1}{n}\sum_{i=1}^nX_{3i}(1-2\cdot W_i-2X_{3i}+4X_{3i}W_i+u_i)$$ As $X_{3i}$ is generated as standard uniform, $\frac{1}{n}\sum_{i=1}^nX_{3i}\to_p 0.5$ and $\frac{1}{n}\sum_{i=1}^nX_{3i}^2\to_p 1/3$. Plugging into the models for men and women then directly yields that the first two entries of $X'y/n$ tend to zero.

Next, $-2\frac{1}{n}\sum_{i=1}^nX_{3i}W_i=-2\frac{1}{n}\sum_{i\in\text{women}}X_{3i}=-2\frac{n_w}{n}\frac{1}{n_w}\sum_{i\in\text{women}}X_{3i}$. The way I generated the data, $\frac{n_w}{n}=0.5$ and, again, $\frac{1}{n_w}\sum_{i\in\text{women}}X_{3i}\to_p0.5$. Similarly, $$4\frac{1}{n}\sum_{i=1}^nX_{3i}^2W_i=4\frac{1}{2}\frac{1}{3}$$ Hence, $$\frac{X'y}{n}\to_p\begin{pmatrix}0\\0\\0\end{pmatrix}$$ Overall, we see that some specific features of the DGP were required to exhibit the result, in particular for the proportion of "men" and "women" in the data as well as for the coefficients and the distribution of the regressors to be such that the OLS estimator has a plim of zero.

There certainly are other possibilities, but playing around with the code will show that yet other specifications will not exhibit this result.

• I also added a little theoretical analysis, if that helps. Jan 30, 2018 at 15:28