Using ANOVA and HSD, I find A and B are not significantly different from C, but I find A is significantly different from B. What does this mean?

Question

I'm learning a bit about ANOVA and Tukey Range test and my understanding is if there's significance between two samples, they don't come from the same distribution. However, in some tests I'm having a hard time understanding what the results imply.

I have three samples named BP, GA, PSO. After collecting results, I run an ANOVA test:

           Df Sum Sq  Mean Sq F value   Pr(>F)    
sample      2 0.4305 0.107626  7.6132 5.87e-06 ***
Residuals 495 6.9977 0.014137     

Since $p < 0.001$, the test shows strong significance that not every sample comes from the same distribution, and with the $F > 1$, confidence in the test. Then, I run Tukey's test to do pairwise analysis:

              diff          lwr         upr     p adj
GA-BP  -0.04500006 -0.091036564 0.001036444 0.0589717
PSO-BP -0.02848048 -0.074516984 0.017556024 0.4387137
PSO-GA  0.01651958 -0.029516924 0.062556084 0.8631683

These results suggest GA and BP come from different distributions, but PSO comes from the same distribution as both BP and GA.

Is there no transitive property here? If A and B come from the same distribution as C, but A and B don't come from the same distribution, what can I infer from this?

Answer 1

It is not unusual to have a significant effect indicating that not all three levels of a factor have the same mean. Then, in post hoc tests, the levels with highest and lowest means will ordinarily be significantly different. But there is no guarantee that you will be able to resolve whether the level with the mean in the middle is significantly different from either of the others.

It seems you have enough power to detect the largest difference, but not enough to be sure about the two smaller differences.

'Not significantly different from' is a long way from 'comes from the same distribution'. The latter is transitive and the former is not. (Transitivity: $A = B$ and $B = C$ imply $A = C.)$ Strictly speaking, your title should not say ?come from the same distribution".