5
$\begingroup$

I'm running 2-way repeated measures ANOVA (3 and 5 levels) with planned contrasts afterwards. I'm interested in comparing

1)levels of the first factor among themselves (e.g. A1B with A2B, A2B with A3B)

2)pairwise comparisons (e.g. A1B1 and A2B4)

Question:

Some sources claim that I need to report effect size for each contrast. I wonder if this makes sense since the design is repeated measures. If so, how the contrasts should be calculated? I don't think the standard formula for Cohen's d works in this case since it doesn't take into account the correlation.

Improve this question
$\endgroup$
8
  • $\begingroup$ How many levels per repeated measures factor? Are your pairwise contrasts just comparing B1 to B2 at level A1 and B1 to B2 at level A2, or do you have something more complex going on? $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 3:09
  • $\begingroup$ Perhaps the answer to this question is tied up in the answer to my first set of questions, but what part of this do you see in the context of a contrast analysis? $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 3:10
  • $\begingroup$ @drknexus I have updated the description of the question. I'm comparing 2 levels of the same factor on average and 2 different pairs (e.g. A1B1 and A2B4) $\endgroup$
    – adelweis
    Commented Jan 30, 2013 at 3:18
  • $\begingroup$ What sort of error term are you thinking about using for your contrasts? Do you think you'll run each as a paired samples t-test, or do you have designs on trying to use the pooled error term. If the later, what does your test of sphericity look like? $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 3:30
  • 1
    $\begingroup$ @drknexus As far as I remember MSerror=SS/df, thus it is also affected. $\endgroup$
    – adelweis
    Commented Jan 30, 2013 at 4:29

2 Answers 2

Reset to default
4
$\begingroup$

Your effect sizes for your pairwise contrasts would be derived from the test that is associated with the effect. For example, if you are doing a pairwise t-test comparing cells of your design, then your effect sizes would be derived from each of those pairwise t-tests (using whatever error term you select for those pairwise tests). As a pairwise t-test is computationally equivalent to a one-sample t-test, you should be able to calculate the Cohen's d using the formula $\bar{d}\over{s_{d}}$ (although this is what is commonly considered Cohen's d, I understand that Cohen's formulas actually reflected the use of $\sigma$).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Do you mean that even though I'm using MSe from 2-way ANOVA to calculate contrasts, when calculating Effect Size I can use SD between two cells instead? $\endgroup$
    – adelweis
    Commented Jan 30, 2013 at 4:37
  • $\begingroup$ You can't use the between subjects variance in each of those cells. It doesn't match the test you are doing. You could use the SD of the differences (difference scores) between the two cells... but only if your contrast was actually calculated as a paired samples t-test using the standard deviation of differences as your error term. $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 5:33
  • $\begingroup$ However, since you are using MSe, I think what you would do is use $\sqrt{MSe}$ as your estimate of $s_{d}$. I tried to spot check my intuition but ezANOVA doesn't like yielding its tasty source table secrets for within subjects designs. Since you have the values in front of you, you can do a quick sanity check on the procedure by comparing $s_{d}$ calculations with non-GG adjusted $\sqrt{MSe}$ values. To the extent the assumption of sphericity is correct, these values should be similar. For your actual calculation you would... I guess... use the GG corrected $MS_{error}$. $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 5:35
  • $\begingroup$ If you wait around a few hours, I'm sure you'll get more skilled and authoritative help than me. $\endgroup$
    – russellpierce
    Commented Jan 30, 2013 at 5:36
  • $\begingroup$ Thanks for your help. This is actually the only thing that seems to me reasonable. However I'm still not sure if using Cohen's d here is a valid approach. $\endgroup$
    – adelweis
    Commented Jan 30, 2013 at 6:30
3
$\begingroup$

If you are doing this using conventional ANOVA, you are making a mistake. This should be done as a mixed-effect repeated measures (MERM) model. The test of sphericity is 30 years out of date. Using mixed-effects models, you define a proper covariance model (which may include the compound symmetry model, but usually does not), and test things properly. These tests are done using PROC MIXED in SAS, also can be done in R or SPSS. Many programs do not do these correctly. If the program warns you about compound symmetry, you are using a bad program. And for those who are warning about compound symmetry and the GG correction, you need to update your methodology. You are out of date. And if you do use MERM, it is easy to specify contrasts. I do this all the time - in SAS. FInally, do not do repeated paired t-tests. Very much the wrong approach. You need to fit the entire design, define contrasts which properly compare between levels, and use the procedure defined error term.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ A little strongly worded. Not all fields are at peace with mixed effect models. Many are far from consensus on how to evaluate effects in regards to NHST. Sticking a bunch of non-orthogonal contrasts into a single model and then interpreting the results in a NHST framework is certain to lead to tears. (Of course there is plenty of room to argue the NHST framework is also outdated... however it still is pretty dominant in some fields). $\endgroup$
    – russellpierce
    Commented Jan 31, 2013 at 1:14
  • $\begingroup$ @russellpierce Could you by any chance point me to bibliography on the (lack of) consensus on what you mention, the problems with non-orthogonal contrasts in NHST? $\endgroup$
    – JEquihua
    Commented Nov 17, 2019 at 20:29
  • $\begingroup$ The problem of non-orthogonal contrasts is just a gedanken experiment. Under, most forms, of NHST you're just testing for the presence/absence of an effect. If your effects are non-orthogonal then you can't be sure about the attribution of the shared variance between two contrasts. As for the lack of consensus, IDK, I wrote that 7 years ago - perhaps a consensus has formed since then (?). I was perhaps most influenced by {lme4} which refused to provide p-values. stat.ethz.ch/pipermail/r-sig-mixed-models/2008q2/000904.html is representative, but far from canonical. $\endgroup$
    – russellpierce
    Commented Nov 18, 2019 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.