Variable is not statistically significant in single variable linear regression but is significant in multiple regression

Question

I had a friend ask me about this question, and I'm not 100% sure about the data she's using due to the sensitivity/privacy of the data (other than it's medical data). Sorry for not providing context, but hopefully you can think of scenarios or conditions in which the following results would happen.

She was telling me that in her analysis, when she tried to model Y just using variable X, it was not statistically significant. However when she modeled Y using both X and Z, both variables were statistically significant. Why/when does this happen? Is it because of the interaction of variables X and Z, even though X alone would not be statistically significant? Would appreciate any insights!

Thank you!

Answer 1

It depends on what you mean by the word interaction, which is a technical term in statistics with a very specific meaning. In that sense, no, it's not necessarily because of an interaction (though it could be). It is very commonly due to simple correlation between your features (variables).

For example, say $Y$ is related to two features $X_1$ and $X_2$ like:

$$ Y = X_1 + X_2 + \epsilon_1 $$

Where $\epsilon$ is some random noise term. If you estimate a linear model with both features, you'll get coefficients of about $1.0$.

Now suppose that $X_1$ and $X_2$ are correlated, say something like:

$$ X_2 = - X_1 + \epsilon_2 $$

Where, again, $\epsilon_2$ is another random noise term. Then the univariate relationship between $Y$ and $X_1$ is:

$$ Y = X_1 + (-X_1 + \epsilon_2) + \epsilon_1 = \epsilon_1 + \epsilon_2 = \text{noise} $$

So if we estimate a model with just X_1, we get a coefficient of about 0.0, probably insignificant.