Recommended effect size measure for repeated measures t-tests and repeated measures contrasts on experimental data Context
This came up recently in a consulting context. A researcher was performing repeated measures t-tests based on experimental data.
Some of the analyses involved comparing one condition with another. Other analyses involved performing contrasts comparing one or more conditions with one or more other conditions.
Question


*

*What effect size measure would you recommend using in relation to (a) repeated measures t-tests (b) repeated measures contrasts comparing one or more group means with one or more other group means?

*If you were reporting a d-based effect size measure, which measure of the within group standard deviation would you use?

*Are there any references that you would recommend?


I have a few thoughts, but I'm keen to get your suggestions.
 A: The answer here depends on your situation.  Dunlap, Cortina, Vaslow, and Burke (1996) argued that the effect size should be calculated using a SD based on pooled variance from separate conditions, as is typical in independent groups studies, even with repeated measurements.  Their argument was that the study may be replicated with a between design and the effect sizes will be more comparable across studies in meta-analysis with that effect size measure. They asserted that the effect size is the effect size and shouldn't be influenced by the correlation in the measurement in a repeated measures design.
Unfortunately, this suggestion has been overgeneralized in some literatures (and in Cortina's book I believe).  When it's not possible to design an experiment any other way than repeated measures then using the between S effects size is a mistake.  It will underestimate the size of the effect and be useless in power calculations.
Imagine an attentional cueing study where you need to study a single mental state, (e.g. oriented in a direction indicated by an arrow), and have to measure the effect comparing performance at the indicated location and one that is not.  That study has to be done within and there is no other way to do it.  In that case, the need to have an effect size comparable to situations where the study is done with independent groups vanishes because the independent group study couldn't occur.  Using the between S effect size would not be a useful estimate in seeing the number of S's to replicate the study while the within would.  The between S effect size would tend to vastly underestimate what you actually needed to measure, which is the effect within.
