In this video, Khan Academy is teaching Chi Squared test with an example: https://www.youtube.com/watch?v=hpWdDmgsIRE

Given 3 medicine, $herb_1$, $herb_2$, and $placebo$, we give these to 3 different groups of people during flu season. For each group, we can see how many people got sick and how many people didn't.

Eg. $$ \begin{array}{c|lcr} & \text{Herb 1} & \text{Herb 2} & \text{Placebo} \\ \hline \text{sick} & 20 & 30 & 30 \\ \text{not sick} & 100 & 110 & 90 \end{array} $$

Khan's null hypothesis is "herbs do nothing". He assumed the null hypothesis, did the chi squared test, it was within bounds, so he accepted the null hypothesis.

My question is: If that "placebo" column wasn't there, no matter how extreme the effect of the herbs were, we would still accept the null hypothesis since we don't have anything else to compare to. So, with just the $Herb_1$ and $Herb_2$ data, is the test meaningless?

Furthermore, if there were, say 100 Herbs, and just 1 placebo group, the amount of data from the herbs would "drown out" the placebo group. In this case, is the test also meaningless?

This is a pretty simple example, but I can imagine in real life some test can be constructed to take advantage of this... Is there a systematic way to not fall into this trap? (Maybe something like running the test assuming $h1$ and see if the results are different?)


1 Answer 1


I think I simply misunderstood how a Hypothesis Test works. We can NEVER accept a null hypothesis; we can only say we can't reject it. In this case we can only say "we can't reject the hypothesis that herbs do nothing."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.