Why not remove an experimental group and do a t-Test instead of an ANOVA?

Question

This has never been made clear to me before, so I would love some help.

Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I hypothesise that either treatment X or Y will have an effect. I collect data and find that ANOVA shows no group effects, however if I just look at A vs B-treatment with t-Test, result is highly significant.

Now here is my question: Is it acceptable to exclude group C from the analysis and conclude that B-treatment has a real effect? If not, why? I understand the issues of multiple testing type 1 errors testing on the same samples, but in this case these are independent groups, so why not just remove one from the hypothesis test? They are biologically independent groups, so isnt it true they are effectively like 2 different experiments (A vs B) and (A vs C)?

---

Thanks for the help everyone! I think I understand:
So the issue with multiple testing is that for each sample A B C, the test states there is a 5% chance of seeing a statistical difference by chance. So we can't do multiple t-tests using sample A for example, because we are basically multiplying this probability for sample A, therefore increasing the false positive error rate. Is this the idea? I’m trying to get a bit of a grip on how the mathematics of multiplying probabilities relates to the biology. So if we wanted to do several t-tests we need several groups (e.g., A1 B1, A2 B2), is that right?

Answer 1

Here is an example in which a t test distinguishes between A and B, but a one-way ANOVA does not find any significant differences. The trouble is that group C has a large variance, which inflates the error, preventing the ANOVA from finding differences.

Descriptive Statistics: A, B, C

Variable   N    Mean  SE Mean  StDev  Minimum      Q1  Median      Q3  Maximum
A         10  114.70     4.14  13.09    95.00  103.00  114.00  126.00   136.00
B         10  101.70     4.14  13.09    82.00   90.00  101.00  113.00   123.00
C         10   104.1     10.8   34.2     37.0    79.5   119.0   129.3    142.0

A two-sample t test finds a significant difference, at the 5% level, between A and B.

Pooled Two-sample T for A vs B

    N   Mean  StDev  SE Mean
A  10  114.7   13.1      4.1
B  10  101.7   13.1      4.1

Difference = μ (A) - μ (B)
T-Test of difference = 0 (vs ≠): 
    T-Value = 2.22  P-Value = 0.039  DF = 18

However, a one-way ANOVA finds no differences. (Because no differences are found it is not appropriate to do ad hoc comparisons. However, if you do Tukey's HSD procedure anyhow, it finds no significant differences with a family error rate of 5%.)

One-way ANOVA: A, B, C 

Method

Null hypothesis         All means are equal
Alternative hypothesis  At least one mean is different
Equal variances were assumed for the analysis.

Analysis of Variance

Source  DF       SS      MS  F-Value  P-Value
Factor   2    957.1   478.5     0.95    0.399
Error   27  13607.1   504.0
Total   29  14564.2

Notes: (1) A Welch ANOVA (not assuming equal variances) also finds no significant differences. The effective Error DF is reduced to about 17 on account of heteroscedasticity. (2) Output is from Minitab, abridged for relevance. (3) Fake normal data with respective population means 115, 100, 105 and population standard deviations 15, 15, 25. So there really are differences, between all pairs of groups, which the ANOVA does not have the power to detect.

Answer 2

Super short answer: because the results of such a t test of A vs. B would actually be a t of (A vs. B | prior rejection of both A vs. C & B vs. C).

How would you communicate the latter mathematically?

How would you appropriately adjust your p values from your single A vs. B test to reflect that conditionality?

Answer 3

The issue of multiple testing does not only apply to dependent samples. You are mentioning that these are basically two different experiments, but that does not matter for the effect of multiple testing. The issue that defines multiple testing is that when using a p-value, you have an explicit statement that 5% of all your experiments will be significant even if they are not.

Let assume for ones that instead of two independent experiments, you do 100 then by definition of the p-value 5 of them would be significant.

Edit: maybe more specific to your example lets say you have more than three conditions, the larger the number of conditions becomes the more likely it is you will find two groups that are significantly different from each other.

You could try to do a post hoc contrast test. SPSS gives the option to do so if you specifically want to compare two groups.