0
$\begingroup$

Beginner to hypothesis testing. Have removed/replaced terminology, but these samples are from real-world populations doing a specific real world action where whichever option they picked splits them (binary choice). Samples are 1000 of each population, population for group 1 - 30100, population for group 2 - 75330.

Sample 1 mean 0.472, std 0.269, var 0.072

Sample 2 mean 0.418, std 0.243, var 0.059

$H_0$ : Sample 1 mean - sample 2 mean > 0

$H_1$ : Sample 1 mean - sample 2 mean <= 0

alpha=0.05, therefore norm.s.inv(0.95)=1.644.

Observed sample means, $0.472-0.418=0.054$

$1.644 \cdot \sqrt{(0.072/1000)+(0.059/1000)} = 0.018$

$0.054$ is greater than $0.018$, therefore there is than 5% chance of getting this difference in sample means assuming the $H_0$, so we reject the null hypothesis.


However, depending who you ask, they'd switch the two hypothesis, so I tried that too, and it's basically exactly the same outcome.

$H_0$ : Sample 1 mean - sample 2 mean <= 0

$H_1$ : Sample 1 mean - sample 2 mean > 0 (note switched hypothesis)

alpha=0.05, therefore norm.s.inv(0.95)=1.644.

Observed sample means, 0.472-0.418=0.054

$1.644 \cdot \sqrt{(0.072/1000)+(0.059/1000)} = 0.018$

$0.054$ is greater than $0.018$, therefore there is than 5% chance of getting this difference in sample means assuming the $H_0$, so we reject the null hypothesis.

So I've rejected both null hypothesis here, which can't possibly be the case. What am I missing when it comes to interpretation of the hypothesis at the end?

Thanks.

$\endgroup$
  • 1
    $\begingroup$ Please provide some indication from where you got these formular, and what kind of test you are trying to calculate. Most importantly, is this test a directed test, or an undirected test? If it is undirected, you are testing H0: mean1 - mean2 != 0. $\endgroup$ – LiKao Nov 20 '18 at 12:38
  • $\begingroup$ I'm trying to test the difference in the means of the samples, however it's possible for $\bar{X}_1 - \bar{X}_2 <= 0$, rather than $\bar{X}_1 - \bar{X}_1 = 0$, which has confused how I write the hypotheses, so I tried reversing them expecting to see the opposite outcome. $\endgroup$ – MB141 Nov 20 '18 at 13:53
  • $\begingroup$ Your first $H_0$ is improper. You could have $H_0: \mu \ge 0$ and $H_1: \mu < 0,$ but not $H_0: \mu > 0$ and $H_1: \mu \le 0,$ The null hypothesis must always contain an $=$-sign, whether as $=, \le,$ or $\ge.$ // You need to decide which of the three possible forms of the null hypothesis is correct in your situation. $\endgroup$ – BruceET Nov 20 '18 at 17:47
1
$\begingroup$

First of all, what you are doing is a Welch's test. The statistic is given by the formula

$t = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{s_1^2}{N_1} + \frac{s_2^2}{N_2}}}$

Clearly, when $\bar{X}_1 < \bar{X}_2$, then $t < 0 < 0.018$.

Think of the two possible tests as of looking at your problem from two sides. But when you look from the other side, you need to switch the signs. Your second formulation becomes:

$H_0: \mu_2 - \mu_1 \le 0$

$H_1: \mu_2 - \mu_1 > 0$

And since $\mu_2 - \mu_1 = -0.054 < 0.018$, you don't reject the $H_0$.

In essence, in a two-sided test ($H_1: \mu_2 - \mu_1 \ne 0$) you are looking for unusually small or unusually large values of the statistic $t$ (given $H_0$), and if you test in one direction only, you are only looking for unusually large values of the statistic $t$. However, if $\mu_2 < \mu_1$, the value of the statistic is smaller than 0.

EDIT: You are not directly calculating the $t$ statistic. Actually, you multiply the p-value threshold ($1.644$) by the denominator ($\sqrt{...}$) of the t statistic and compare it to the numerator ($\mu_1 - \mu_2$). But see what happens if you divide both sides by $\sqrt{(0.072/1000)+(0.059/1000)}$. On the left hand, you have $1.644$. On the right hand, you have

$\frac{0.472-0.418}{\sqrt{(0.072/1000)+(0.059/1000)}} = \frac{0.054}{0.011}=4.718$, which is the $t$ statistic from the Welch test.

Another note, however. I have now googled the norm.s.inv function (which appears to be an Excel function) and it is definitely the wrong one to use. Please just use the Welch test or use the T.INV function (if I google correctly). Explanation:

The $t$ statistic is not normally distributed. It has a $t$ distribution with a certain number of degrees of freedom. In a Welch test, the number of degrees of freedom is approximated as follows:

$\frac{(\frac{s_1^2}{N_1}+\frac{s_2^2}{N_2})^2}{\frac{s_1^4}{N_1^2(N_1-1)} + \frac{s_2^4}{N_2^2(N_2-1)}}$

What the person presenting the Khan talk did was to approximate the distribution using normal distribution, probably to make the talk simpler. However, you should not do that with real data.

This looks complicated, but really isn't. You will find a nice explanation here.

EDIT #2 Another take.

Think of t-test as capable only of doing one of two things:

  • test whether two means, say $\mu_1$ and $\mu_2$ are different, or
  • test whether $\mu_1$ is larger than $\mu_2$.

Note that you cannot directly test whether $\mu_1$ is smaller than $\mu_2$, but you are allowed to rename them or switch their places. Switching their places is the same as changing the sign of the $t$ you calculated, because $a-b = -(b-a)$.

$\endgroup$
  • $\begingroup$ I'm following the method found here - I thought what I was doing was the same as what he was doing and the same method would work. Welch's test returns t=4.71 which is so far away from any expected result I have no idea how to interpret it. Can you clarify why this is a Welch's test? $\endgroup$ – MB141 Nov 20 '18 at 13:50
  • $\begingroup$ Because the formula you are using is a Welch's t-test formula. Note that you are not calculating the t directly in your formula. I will add an edit. $\endgroup$ – January Nov 20 '18 at 13:51
  • $\begingroup$ @MichaelBrown The page you mentioned does indeed give the procedure for the two-sided (undirected) t-test (Welch). One-sided or undirected means, you are just asking "H1: Could this two samples be different" or rather "H0: Could these two samples be the same". There is no hypothesis with ">" when you use this version. Or on the page "Hypothesis Test for Difference of Means", so either "mean1 > mean2" or "mean2 > mean1". Now if I switch this, I get "mean2 > mean1" or "mean1 > mean2", which is the same. January shows how to turn this into a directed (one-sided) test. $\endgroup$ – LiKao Nov 20 '18 at 13:58
  • $\begingroup$ @MichaelBrown as you can see now, the numeric result is similar to what you got. $\endgroup$ – January Nov 20 '18 at 13:59
  • 1
    $\begingroup$ @MichaelBrown The way you calculated it, your H0 is not $\mu_1 -\mu_2 > 0$. It is $\mu_1 - \mu_2 = 0$. And your H1 is $\mu_1 - \mu_2 \neq 0$ and not (as you assume) $\mu_1 - \mu_2 \leq 0$. Now to show what this means, H1 is equivalent to $\mu_1 - \mu_2 > 0$ or $\mu_2 - \mu_1 > 0$. If you switch the signs, in both parts of this "or" you get the same thing. $\endgroup$ – LiKao Nov 20 '18 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.