This has never been made clear to me before, so I would love some help.

Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I hypothesise that either treatment X or Y will have an effect. I collect data and find that ANOVA shows no group effects, however if I just look at A vs B-treatment with t-Test, result is highly significant.

Now here is my question: Is it acceptable to exclude group C from the analysis and conclude that B-treatment has a real effect? If not, why? I understand the issues of multiple testing type 1 errors testing on the same samples, but in this case these are independent groups, so why not just remove one from the hypothesis test? They are biologically independent groups, so isnt it true they are effectively like 2 different experiments (A vs B) and (A vs C)?

Thanks for the help everyone! I think I understand:
So the issue with multiple testing is that for each sample A B C, the test states there is a 5% chance of seeing a statistical difference by chance. So we can't do multiple t-tests using sample A for example, because we are basically multiplying this probability for sample A, therefore increasing the false positive error rate. Is this the idea? I’m trying to get a bit of a grip on how the mathematics of multiplying probabilities relates to the biology. So if we wanted to do several t-tests we need several groups (e.g., A1 B1, A2 B2), is that right?

  • 1
    $\begingroup$ Why did not do two groups (A vs B) experiment? You have too much money so the group C was added? $\endgroup$ – user158565 Dec 5 '18 at 5:31
  • $\begingroup$ @user158565 Group C was part of the original stated hypothesis, however results not significant by ANOVA analysis including A and B. So my question is why not just remove C? It’s a biologically independent group, so I don’t see the problem... $\endgroup$ – AmielMatt. Dec 5 '18 at 5:38
  • 3
    $\begingroup$ Statistical analysis (methods) is a set of rule that is decided before the experiment. After experiment, just follow the rules and cannot change the rules. $\endgroup$ – user158565 Dec 5 '18 at 5:54
  • 1
    $\begingroup$ If all assumptions for a one-way ANOVA with three levels A, B, C are met, it would be unusual (not impossible) to have the ANOVA find no significant differences among A, B, C, but to have t-tests of A vs B, etc. show significant differences. Maybe data are not normal or groups have different variances. Did you check that? // Are sample sizes of the three groups about the same? What are the p-values of the tests? If you can you provide, sample sizes, sample means and sample standard deviations for each of the three groups, someone might be able to give a data-based explanation. $\endgroup$ – BruceET Dec 5 '18 at 9:39

Here is an example in which a t test distinguishes between A and B, but a one-way ANOVA does not find any significant differences. The trouble is that group C has a large variance, which inflates the error, preventing the ANOVA from finding differences.

Descriptive Statistics: A, B, C

Variable   N    Mean  SE Mean  StDev  Minimum      Q1  Median      Q3  Maximum
A         10  114.70     4.14  13.09    95.00  103.00  114.00  126.00   136.00
B         10  101.70     4.14  13.09    82.00   90.00  101.00  113.00   123.00
C         10   104.1     10.8   34.2     37.0    79.5   119.0   129.3    142.0

enter image description here

A two-sample t test finds a significant difference, at the 5% level, between A and B.

Pooled Two-sample T for A vs B

    N   Mean  StDev  SE Mean
A  10  114.7   13.1      4.1
B  10  101.7   13.1      4.1

Difference = μ (A) - μ (B)
T-Test of difference = 0 (vs ≠): 
    T-Value = 2.22  P-Value = 0.039  DF = 18

However, a one-way ANOVA finds no differences. (Because no differences are found it is not appropriate to do ad hoc comparisons. However, if you do Tukey's HSD procedure anyhow, it finds no significant differences with a family error rate of 5%.)

One-way ANOVA: A, B, C 


Null hypothesis         All means are equal
Alternative hypothesis  At least one mean is different
Equal variances were assumed for the analysis.

Analysis of Variance

Source  DF       SS      MS  F-Value  P-Value
Factor   2    957.1   478.5     0.95    0.399
Error   27  13607.1   504.0
Total   29  14564.2

Notes: (1) A Welch ANOVA (not assuming equal variances) also finds no significant differences. The effective Error DF is reduced to about 17 on account of heteroscedasticity. (2) Output is from Minitab, abridged for relevance. (3) Fake normal data with respective population means 115, 100, 105 and population standard deviations 15, 15, 25. So there really are differences, between all pairs of groups, which the ANOVA does not have the power to detect.


Super short answer: because the results of such a t test of A vs. B would actually be a t of (A vs. B | prior rejection of both A vs. C & B vs. C).

How would you communicate the latter mathematically?

How would you appropriately adjust your p values from your single A vs. B test to reflect that conditionality?

  • 1
    $\begingroup$ Not sure why the downvote, but a perusal of my answers will reveal that I am receptive to explicit criticism in editing my answers. $\endgroup$ – Alexis Jan 5 at 17:02

The issue of multiple testing does not only apply to dependent samples. You are mentioning that these are basically two different experiments, but that does not matter for the effect of multiple testing. The issue that defines multiple testing is that when using a p-value, you have an explicit statement that 5% of all your experiments will be significant even if they are not.

Let assume for ones that instead of two independent experiments, you do 100 then by definition of the p-value 5 of them would be significant.

Edit: maybe more specific to your example lets say you have more than three conditions, the larger the number of conditions becomes the more likely it is you will find two groups that are significantly different from each other.

You could try to do a post hoc contrast test. SPSS gives the option to do so if you specifically want to compare two groups.


Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.