4
$\begingroup$

I have a data set containing three groups (A, B, C). Group A and Group B have undergone two different types of training designed to improve the same thing. Group C is my control group with no training. I have specific hypotheses stating that:

  • Group A will be better than group C
  • Group B will be better than group C and
  • Group B will be better than group group A.

This is because the training used in group A has already been shown to improve over control (so I am trying to replicate that effect), while group B is a new training method which I expect to exceed both groups A and C.

How would I go about conducting this analysis? I have conducted a one-way ANOVA with planned comparisons, but I realized that my comparisons would have to be non-orthogonal since comparing B to A and C in one contrast would be pooling variance from an experimental and control condition together. Optimally, I want these comparisons:

 A   B   C

 1   0  -1
 0   1  -1
-1   1   0

What is the proper way to conduct such comparisons?

Improve this question
$\endgroup$
  • $\begingroup$ The 2 comparisons among means that you stated in English do not agree with the 3 comparisons among means implied by your matrix of contrasts. If I understand correctly, you say that you want to compare A to C, and then B to the mean of A and C. But the contrasts that you wrote imply that you want to test all pair-wise comparisons. (The 3 contrasts that you wrote actually form a linearly dependent set, so they wouldn't work in an actual regression, but I take them to mean that you are interested in the pairwise comparisons.) So which set of questions is it that you're interested in? $\endgroup$ – Jake Westfall Oct 3 '13 at 3:33
  • $\begingroup$ I'm sorry about not being clear, I can see now that the English statement was a bit ambiguous. In fact you are correct, I want to test all pair-wise comparisons. $\endgroup$ – JamesC Oct 3 '13 at 10:49
4
$\begingroup$

You cannot test all 3 pairwise comparisons within a single model because it will always be the case that one of the codes is a perfect linear combination of the other 2 codes. For example, in the codes you wrote, if we call the rows/contrasts $C_1$, $C_2$ and $C_3$ (from top row to bottom row), notice that $C_1 = C_2 - C_3$.

On a more intuitive level, we know that something would have to be wrong with a model that only made predictions based on the 3 pairwise differences, because for any given set of 3 pairwise differences, there are infinitely many sets of 3 group means that would produce those 3 group differences. For example, if the differences are, $\bar{A}-\bar{B}=1$, $\bar{B}-\bar{C}=1$, and $\bar{A}-\bar{C}=2$, then the group means could be $\bar{A}=2,\bar{B}=1,\bar{C}=0$, or $\bar{A}=3,\bar{B}=2,\bar{C}=1$, or $...$.

Probably the easiest way to test all pairwise difference is just to fit 2 separate models, one which compares A to B and A to C (e.g., dummy codes with A as the reference category), and then a separate model that has a code comparing B to C.

As for correcting $\alpha$, for tests of all pairwise differences among a set of means, a conventional method is to use Tukey's Honestly Significant Difference (HSD) procedure. Many resources on doing this computation exist; the treatment in this document looks helpful as it shows the formula for the test statistic in the unequal sample sizes case. Tables of critical values for the test statistic can be found in various places, e.g. here.

Improve this answer
$\endgroup$
  • $\begingroup$ Jake, even though this answer was accepted by the OP, I find that is does not really answer the question as it was formulated. You explain that to test all three pairwise contrasts, one linear model is not enough. However, what seems to be the main question, is how to account for multiple comparisons. We can even think of doing three separate t-tests, i.e. fit not 2, but 3 models (with or without using the pooled within-group variance from ANOVA). But how to adjust the $\alpha$ for multiple comparisons? Is it important that contrasts are non-orthogonal? $\endgroup$ – amoeba Dec 3 '15 at 16:15
  • $\begingroup$ @amoeba Yeah, I suppose that's right. The OP put correcting for alpha in the title, but then never actually mentions this again in the body of the question, which is I guess how I missed it. I'll put in a little edit about Tukey's HSD. $\endgroup$ – Jake Westfall Dec 3 '15 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.