8
$\begingroup$

I did a one-way ANOVA followed by a Tukey's test to compare the means of different treatments.

Let's say the treatments are A, B and C.

The table of multiple comparisons tells me there is a significant difference between B and C. However, these two are not significantly different from A, and therefore there are in the same subset when we order the results.

Can I say there is a significant difference between B and C, or is that not possible?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
12
$\begingroup$

Your discomfort is related to the theory of vagueness in philosophy. Statisticians generally believe that cases like yours are resolvable, and thus, this situation is a case of ambiguity rather than true vagueness (although this is ultimately a philosophical belief rather than something that can be proven). So, from a statistical perspective, we say that you simply have insufficient power, as standard logic (crisp sets) demands that A is either drawn from the same population as B, as C, or is drawn from it's own population. Thus, you must have at least 1 type II error. That is, if A is drawn from the same population as B, then $\text H:A=C$ should have been rejected, likewise if it's the same as C, then $\text H:A=B$ should have been rejected, and if drawn from a distinct population, then both nulls should have been rejected. In the interim, you can say that there is a difference between B and C. (Sorry to get all metaphysical.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @gung for the reference to theory of vagueness (+1). $\endgroup$
    – gui11aume
    Jul 23, 2012 at 22:16
  • 1
    $\begingroup$ +1 Perhaps a little more relevant is the prototype theory of cognitive science, which provides some empirically justified explanations of how this kind of vagueness not only arises but actually is commonplace. $\endgroup$
    – whuber ♦
    Jul 23, 2012 at 22:36
  • $\begingroup$ thank you for your helpful answer and no need to apologize to get all metaphysical.. Think it's great to be able to do that and according to me science needs it a bit more! :) $\endgroup$
    – gips
    Jul 24, 2012 at 8:13
6
$\begingroup$

Most statisticians would conclude that there is a significant difference between B and C in that case.

It is generally agreed that rejecting a null hypothesis is more informative than accepting it. As a student, I even had to write "the null hypothesis is not rejected" instead of "the null hypothesis is accepted". This is so because you know that the probability of being wrong while rejecting is small (typically lower than 0.05) but you have no idea of the probability of being wrong while accepting (which can be close to 1).

For the rest, I fully agree with the answer of @gung.

Improve this answer
$\endgroup$
1
  • $\begingroup$ ok, thank you! and i actually never thought about why I couldnt say we accept the null hypothesis. thx for making that clearer aswell! $\endgroup$
    – gips
    Jul 24, 2012 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.