# Effect size in contrast analysis

I'm running 2-way repeated measures ANOVA (3 and 5 levels) with planned contrasts afterwards. I'm interested in comparing

1)levels of the first factor among themselves (e.g. A1B with A2B, A2B with A3B)

2)pairwise comparisons (e.g. A1B1 and A2B4)

Question:

Some sources claim that I need to report effect size for each contrast. I wonder if this makes sense since the design is repeated measures. If so, how the contrasts should be calculated? I don't think the standard formula for Cohen's d works in this case since it doesn't take into account the correlation.

• How many levels per repeated measures factor? Are your pairwise contrasts just comparing B1 to B2 at level A1 and B1 to B2 at level A2, or do you have something more complex going on? Commented Jan 30, 2013 at 3:09
• Perhaps the answer to this question is tied up in the answer to my first set of questions, but what part of this do you see in the context of a contrast analysis? Commented Jan 30, 2013 at 3:10
• @drknexus I have updated the description of the question. I'm comparing 2 levels of the same factor on average and 2 different pairs (e.g. A1B1 and A2B4) Commented Jan 30, 2013 at 3:18
• What sort of error term are you thinking about using for your contrasts? Do you think you'll run each as a paired samples t-test, or do you have designs on trying to use the pooled error term. If the later, what does your test of sphericity look like? Commented Jan 30, 2013 at 3:30
• @drknexus As far as I remember MSerror=SS/df, thus it is also affected. Commented Jan 30, 2013 at 4:29

Your effect sizes for your pairwise contrasts would be derived from the test that is associated with the effect. For example, if you are doing a pairwise t-test comparing cells of your design, then your effect sizes would be derived from each of those pairwise t-tests (using whatever error term you select for those pairwise tests). As a pairwise t-test is computationally equivalent to a one-sample t-test, you should be able to calculate the Cohen's d using the formula $\bar{d}\over{s_{d}}$ (although this is what is commonly considered Cohen's d, I understand that Cohen's formulas actually reflected the use of $\sigma$).

• Do you mean that even though I'm using MSe from 2-way ANOVA to calculate contrasts, when calculating Effect Size I can use SD between two cells instead? Commented Jan 30, 2013 at 4:37
• You can't use the between subjects variance in each of those cells. It doesn't match the test you are doing. You could use the SD of the differences (difference scores) between the two cells... but only if your contrast was actually calculated as a paired samples t-test using the standard deviation of differences as your error term. Commented Jan 30, 2013 at 5:33
• However, since you are using MSe, I think what you would do is use $\sqrt{MSe}$ as your estimate of $s_{d}$. I tried to spot check my intuition but ezANOVA doesn't like yielding its tasty source table secrets for within subjects designs. Since you have the values in front of you, you can do a quick sanity check on the procedure by comparing $s_{d}$ calculations with non-GG adjusted $\sqrt{MSe}$ values. To the extent the assumption of sphericity is correct, these values should be similar. For your actual calculation you would... I guess... use the GG corrected $MS_{error}$. Commented Jan 30, 2013 at 5:35
• If you wait around a few hours, I'm sure you'll get more skilled and authoritative help than me. Commented Jan 30, 2013 at 5:36
• Thanks for your help. This is actually the only thing that seems to me reasonable. However I'm still not sure if using Cohen's d here is a valid approach. Commented Jan 30, 2013 at 6:30

If you are doing this using conventional ANOVA, you are making a mistake. This should be done as a mixed-effect repeated measures (MERM) model. The test of sphericity is 30 years out of date. Using mixed-effects models, you define a proper covariance model (which may include the compound symmetry model, but usually does not), and test things properly. These tests are done using PROC MIXED in SAS, also can be done in R or SPSS. Many programs do not do these correctly. If the program warns you about compound symmetry, you are using a bad program. And for those who are warning about compound symmetry and the GG correction, you need to update your methodology. You are out of date. And if you do use MERM, it is easy to specify contrasts. I do this all the time - in SAS. FInally, do not do repeated paired t-tests. Very much the wrong approach. You need to fit the entire design, define contrasts which properly compare between levels, and use the procedure defined error term.

• A little strongly worded. Not all fields are at peace with mixed effect models. Many are far from consensus on how to evaluate effects in regards to NHST. Sticking a bunch of non-orthogonal contrasts into a single model and then interpreting the results in a NHST framework is certain to lead to tears. (Of course there is plenty of room to argue the NHST framework is also outdated... however it still is pretty dominant in some fields). Commented Jan 31, 2013 at 1:14
• @russellpierce Could you by any chance point me to bibliography on the (lack of) consensus on what you mention, the problems with non-orthogonal contrasts in NHST? Commented Nov 17, 2019 at 20:29
• The problem of non-orthogonal contrasts is just a gedanken experiment. Under, most forms, of NHST you're just testing for the presence/absence of an effect. If your effects are non-orthogonal then you can't be sure about the attribution of the shared variance between two contrasts. As for the lack of consensus, IDK, I wrote that 7 years ago - perhaps a consensus has formed since then (?). I was perhaps most influenced by {lme4} which refused to provide p-values. stat.ethz.ch/pipermail/r-sig-mixed-models/2008q2/000904.html is representative, but far from canonical. Commented Nov 18, 2019 at 14:59