In this video, Khan Academy is teaching Chi Squared test with an example: https://www.youtube.com/watch?v=hpWdDmgsIRE
Given 3 medicine, $herb_1$, $herb_2$, and $placebo$, we give these to 3 different groups of people during flu season. For each group, we can see how many people got sick and how many people didn't.
Eg. $$ \begin{array}{c|lcr} & \text{Herb 1} & \text{Herb 2} & \text{Placebo} \\ \hline \text{sick} & 20 & 30 & 30 \\ \text{not sick} & 100 & 110 & 90 \end{array} $$
Khan's null hypothesis is "herbs do nothing". He assumed the null hypothesis, did the chi squared test, it was within bounds, so he accepted the null hypothesis.
My question is: If that "placebo" column wasn't there, no matter how extreme the effect of the herbs were, we would still accept the null hypothesis since we don't have anything else to compare to. So, with just the $Herb_1$ and $Herb_2$ data, is the test meaningless?
Furthermore, if there were, say 100 Herbs, and just 1 placebo group, the amount of data from the herbs would "drown out" the placebo group. In this case, is the test also meaningless?
This is a pretty simple example, but I can imagine in real life some test can be constructed to take advantage of this... Is there a systematic way to not fall into this trap? (Maybe something like running the test assuming $h1$ and see if the results are different?)
