Question about the logic of hypothesis testing

Question

Let us say that we have this following problem:

"A government agency claims that more than 50% of US tax returns were filed electronically last year. A random sample of 150 tax returns for last year contained 86 that were filed electronically. Test the claim at $\alpha = 0.05$ significance level."

In this problem our null hypothesis is: $H_0 : p \leq 0.50$ and $H_a : p > 0.50$. I've calculated the sample proportion : $\frac{x}{n} = \frac{86}{150} = 0.573$ and the standard error: $\sqrt{\frac{p(1-p)}{n}} = 0.0408$. Now my z-score is 1.79 which is greater than $z=1.64$, so I'm "required to reject the null hypothesis." But my question is how do I know that 86 out of 150 is not an unusual sampling to begin with? Why can't we reject the random sampling and support the claim given by $H_0$?

My last question deals with what to do with $H_a$. Once $H_0$ has been rejected do we say that there is "sufficient evidence for $H_a$" or do we say that "$H_a$ is true but under these conditions :"?

Answer 1

The whole idea behind hypothesis testing is that we never really know with certainty that $H_a$ or $H_0$ is true. We only have evidence in support of a conclusion, evidence which carries some intrinsic (and presumably random) sampling error. Thus, there is always some probability that we could have observed a sample that, according to our test statistic, would indicate rejecting $H_0$, when the true state of affairs is otherwise. This probability is what we call Type I error: erroneously rejecting the null hypothesis and stating that we observed an effect, when in reality, it was purely by random chance (natural sampling variation) that we got the data we did.

To give you a more concrete example, suppose I give you a coin. I know the precise probability $p$ that the coin will show heads when tossed, but I don't tell you. It looks fair to you, so suppose you want to test the hypothesis that $H_0 : p = 0.5$ versus $H_a : p \ne 0.5$. That is, you have no reason to suspect it is biased, but if it is in fact biased, you want any empirical evidence of biasedness to be demonstrated with a high degree of confidence. In other words, if you conclude $p \ne 0.5$, you want the probability that you made this conclusion erroneously to be very small, say, no more than $\alpha = 0.01$. This is the maximum Type I error you are willing to tolerate.

Now, to test your hypothesis, you flip the coin 8 times and record the number of heads obtained, $X$. Then $X \sim {\rm Binomial}(8,p)$, where $p$ is unknown. But under the null hypothesis, we would expect $X \sim {\rm Binomial}(8, 0.5)$. Suppose you observed $X = 7$ heads. Is this enough evidence to suggest the coin is biased, such that if you made this conclusion, you would be wrong no more than 1% of the time?

To figure this out, we calculate the probability that $X \ge 7$ or $X \le 1$, given $H_0 : p = 0.5$ (two-sided test). This is $9/128 \approx 0.070 > 0.01$, so this means we have insufficient evidence. In other words, if the coin were in fact fair, we would expect to obtain by random chance an outcome as extreme as 7 out of 8 heads as often as 7% of the time.

Now, suppose you flip the coin $20$ times and get $3$ heads. Is this enough evidence to suggest the coin is biased? Again, for $\alpha = 0.01$, you can calculate the $p$-value to get $\frac{1351}{524288} \approx 0.00257683 < 0.01$, so yes, we do conclude that the coin is biased with 99% confidence: in other words, you could be wrong and the coin is in fact fair (and you just happened to see this outcome by random chance), but the chance of this happening is less than 1%.