# How to interpret a 1% significant relation when adding a control diminishes the significance level at 5%?

In rough terms, I am given a t-statistic of 3.00 of the coefficient of variable $X$, $\beta_1$, after regressing the independent variable $X$ on the dependent $Y$:

$$Y = \alpha + \beta_1 X.$$

I have previously formed the null hypothesis that $X$ explains $Y$. The normal interpretation would be:

"At the significance level of 1% I cannot reject the null hypothesis that $X$ explains $Y$".

However, adding some controls in different multiple regressions, say $Z_1$ in one specification and $Z_2$ in another, the t-statistic of $\beta_1$ drops to roughly 2.00, significant at 5% only, for only a couple of the tested control specifications.

What do I say then for interpreting the null hypothesis?

"I cannot reject the null hypothesis that $X$ explains $Y$ at the significant level %1 or 5%?

I understand that the null hypothesis considers one specification only, but what is the common approach?

In a study, consider I form the variables based on answers on questions of a specific group of people. I then run $$Y=\alpha+\beta_1X ~~~~~H_0: \beta_1=0$$ and $$Y=\alpha+\beta_1X+\gamma Z ~~~~~H_0: \beta_1=0$$

Next day, as an added robustness check and assuming nothing should/has change/changed regarding the answers for forming the variables $Y$, $X$ and $Z$, I ask the same group of people the same questions, and I run the same regression on the new dataset. Now, $\beta_1$ of the simple model is still significant at 1%, but it is significant at 5% only for the multiple regression model.

How do I treat this case? In rough terms, the two dataset should give the same results. But now, the new dataset is not exactly replicating the result of the former dataset, in terms of statistical significance. How do I interpret this result in my study as a concluding sentence? I say "I cannot reject the null hypothesis that $X$ explains $Y$ at the significant level %1 or 5%?

• What is the new question asking? Is it wondering why two models can get two different p-values or why two datasets can get different p-values? (And is there any mystery at all in either case?)
– whuber
Aug 19 '16 at 20:33
• I'm a bit confused on your added part. Are you now asking why a new dataset isn't replicating the result?
– VCG
Aug 19 '16 at 20:34
• @frage_man Clarified it, thanks, and sorry for my seemingly silly questions, but I have no experience on how people deal with such subtle differences. Aug 19 '16 at 20:38
• No worries just trying to help.
– VCG
Aug 19 '16 at 20:39
– VCG
Aug 19 '16 at 20:52

Initially you estimated:

$Y=\alpha+\beta_1X~~~~~H_0: \beta_1=0$

Then you accept or reject this at some confidence level (you were able to do that at 99%)

Now:

$Y=\alpha+\beta_1X+\gamma Z ~~~~~H_0: \beta_1=0$

Now you fail to reject at the 99% level and you reject at the 95% level.

If those extra Zs belong in the true model, then you ignore the first regression as it was biased. The regression that includes more relevant variables will result in less omitted variable bias and you can have more confidence in the results.

Before you likely had a higher chance of type 1 errors because of omitted variable bias, while the newer specification is a better model.

To answer your added part: If you add extra data points and your results change, then you should maybe estimate the model for all the available data using the fuller model and then determine whether you reject or accept at 1% or 5%.

So suppose you reran the full regression on the whole dataset and now you can only reject at the 5%. You would say that you reject the null that X is unrelated to Y, conditional on all the other variables, at the 95% level.

• I get what you're saying, and largely agree. It strikes me, though, that the null hypothesis has changed. That is because the meaning of "$\beta_1$" has changed. In the first model it's the coefficient of $X$, whereas in the second one it's the coefficient of the residual of $X$ after regressing it on $Z$.
– whuber
Aug 19 '16 at 20:29
• Thank you! I added a clarification in the question, where as an additional robustness check I form a new dataset that should on principle be the same with the first. Aug 19 '16 at 20:29
• @whuber Yes - $\beta_1$ is now the marginal effect of the variation in X orthogonal to Z that is affecting Y.
– VCG
Aug 19 '16 at 20:32