This has never been made clear to me before, so I would love some help.
Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I hypothesise that either treatment X or Y will have an effect. I collect data and find that ANOVA shows no group effects, however if I just look at A vs B-treatment with t-Test, result is highly significant.
Now here is my question: Is it acceptable to exclude group C from the analysis and conclude that B-treatment has a real effect? If not, why? I understand the issues of multiple testing type 1 errors testing on the same samples, but in this case these are independent groups, so why not just remove one from the hypothesis test? They are biologically independent groups, so isnt it true they are effectively like 2 different experiments (A vs B) and (A vs C)?
Thanks for the help everyone! I think I understand:
So the issue with multiple testing is that for each sample A B C, the test states there is a 5% chance of seeing a statistical difference by chance. So we can't do multiple t-tests using sample A for example, because we are basically multiplying this probability for sample A, therefore increasing the false positive error rate. Is this the idea? I’m trying to get a bit of a grip on how the mathematics of multiplying probabilities relates to the biology. So if we wanted to do several t-tests we need several groups (e.g., A1 B1, A2 B2), is that right?