Hypothesis Testing - Switch hypothesis and get same result?

Question

Beginner to hypothesis testing. Have removed/replaced terminology, but these samples are from real-world populations doing a specific real world action where whichever option they picked splits them (binary choice). Samples are 1000 of each population, population for group 1 - 30100, population for group 2 - 75330.

Sample 1 mean 0.472, std 0.269, var 0.072

Sample 2 mean 0.418, std 0.243, var 0.059

$H_0$ : Sample 1 mean - sample 2 mean > 0

$H_1$ : Sample 1 mean - sample 2 mean <= 0

alpha=0.05, therefore norm.s.inv(0.95)=1.644.

Observed sample means, $0.472-0.418=0.054$

$1.644 \cdot \sqrt{(0.072/1000)+(0.059/1000)} = 0.018$

$0.054$ is greater than $0.018$, therefore there is than 5% chance of getting this difference in sample means assuming the $H_0$, so we reject the null hypothesis.

---

However, depending who you ask, they'd switch the two hypothesis, so I tried that too, and it's basically exactly the same outcome.

$H_0$ : Sample 1 mean - sample 2 mean <= 0

$H_1$ : Sample 1 mean - sample 2 mean > 0 (note switched hypothesis)

alpha=0.05, therefore norm.s.inv(0.95)=1.644.

Observed sample means, 0.472-0.418=0.054

$1.644 \cdot \sqrt{(0.072/1000)+(0.059/1000)} = 0.018$

$0.054$ is greater than $0.018$, therefore there is than 5% chance of getting this difference in sample means assuming the $H_0$, so we reject the null hypothesis.

So I've rejected both null hypothesis here, which can't possibly be the case. What am I missing when it comes to interpretation of the hypothesis at the end?

Thanks.

Answer 1

First of all, what you are doing is a Welch's test. The statistic is given by the formula

$t = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{s_1^2}{N_1} + \frac{s_2^2}{N_2}}}$

Clearly, when $\bar{X}_1 < \bar{X}_2$, then $t < 0 < 0.018$.

Think of the two possible tests as of looking at your problem from two sides. But when you look from the other side, you need to switch the signs. Your second formulation becomes:

$H_0: \mu_2 - \mu_1 \le 0$

$H_1: \mu_2 - \mu_1 > 0$

And since $\mu_2 - \mu_1 = -0.054 < 0.018$, you don't reject the $H_0$.

In essence, in a two-sided test ($H_1: \mu_2 - \mu_1 \ne 0$) you are looking for unusually small or unusually large values of the statistic $t$ (given $H_0$), and if you test in one direction only, you are only looking for unusually large values of the statistic $t$. However, if $\mu_2 < \mu_1$, the value of the statistic is smaller than 0.

EDIT: You are not directly calculating the $t$ statistic. Actually, you multiply the p-value threshold ($1.644$) by the denominator ($\sqrt{...}$) of the t statistic and compare it to the numerator ($\mu_1 - \mu_2$). But see what happens if you divide both sides by $\sqrt{(0.072/1000)+(0.059/1000)}$. On the left hand, you have $1.644$. On the right hand, you have

$\frac{0.472-0.418}{\sqrt{(0.072/1000)+(0.059/1000)}} = \frac{0.054}{0.011}=4.718$, which is the $t$ statistic from the Welch test.

Another note, however. I have now googled the norm.s.inv function (which appears to be an Excel function) and it is definitely the wrong one to use. Please just use the Welch test or use the T.INV function (if I google correctly). Explanation:

The $t$ statistic is not normally distributed. It has a $t$ distribution with a certain number of degrees of freedom. In a Welch test, the number of degrees of freedom is approximated as follows:

$\frac{(\frac{s_1^2}{N_1}+\frac{s_2^2}{N_2})^2}{\frac{s_1^4}{N_1^2(N_1-1)} + \frac{s_2^4}{N_2^2(N_2-1)}}$

What the person presenting the Khan talk did was to approximate the distribution using normal distribution, probably to make the talk simpler. However, you should not do that with real data.

This looks complicated, but really isn't. You will find a nice explanation here.

EDIT #2 Another take.

Think of t-test as capable only of doing one of two things:

• test whether two means, say $\mu_1$ and $\mu_2$ are different, or
• test whether $\mu_1$ is larger than $\mu_2$.

Note that you cannot directly test whether $\mu_1$ is smaller than $\mu_2$, but you are allowed to rename them or switch their places. Switching their places is the same as changing the sign of the $t$ you calculated, because $a-b = -(b-a)$.