Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I did a one-way ANOVA followed by a Tukey's test to compare the means of different treatments.

Let's say the treatments are A, B and C.

The table of multiple comparisons tells me there is a significant difference between B and C. However, these two are not significantly different from A, and therefore there are in the same subset when we order the results.

Can I say there is a significant difference between B and C, or is that not possible?

share|improve this question
up vote 11 down vote accepted

Your discomfort is related to the theory of vagueness in philosophy. Statisticians generally believe that cases like yours are resolvable, and thus, this situation is a case of ambiguity rather than true vagueness (although this is ultimately a philosophical belief rather than something that can be proven). So, from a statistical perspective, we say that you simply have insufficient power, as standard logic (crisp sets) demands that A is either drawn from the same population as B, as C, or is drawn from it's own population. Thus, you must have at least 1 type II error. That is, if A is drawn from the same population as B, then $\text H:A=C$ should have been rejected, likewise if it's the same as C, then $\text H:A=B$ should have been rejected, and if drawn from a distinct population, then both nulls should have been rejected. In the interim, you can say that there is a difference between B and C. (Sorry to get all metaphysical.)

share|improve this answer
Thanks @gung for the reference to theory of vagueness (+1). – gui11aume Jul 23 '12 at 22:16
+1 Perhaps a little more relevant is the prototype theory of cognitive science, which provides some empirically justified explanations of how this kind of vagueness not only arises but actually is commonplace. – whuber Jul 23 '12 at 22:36
thank you for your helpful answer and no need to apologize to get all metaphysical.. Think it's great to be able to do that and according to me science needs it a bit more! :) – gips Jul 24 '12 at 8:13

Most statisticians would conclude that there is a significant difference between B and C in that case.

It is generally agreed that rejecting a null hypothesis is more informative than accepting it. As a student, I even had to write "the null hypothesis is not rejected" instead of "the null hypothesis is accepted". This is so because you know that the probability of being wrong while rejecting is small (typically lower than 0.05) but you have no idea of the probability of being wrong while accepting (which can be close to 1).

For the rest, I fully agree with the answer of @gung.

share|improve this answer
ok, thank you! and i actually never thought about why I couldnt say we accept the null hypothesis. thx for making that clearer aswell! – gips Jul 24 '12 at 8:13

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.