5
$\begingroup$

Context

This came up recently in a consulting context. A researcher was performing repeated measures t-tests based on experimental data. Some of the analyses involved comparing one condition with another. Other analyses involved performing contrasts comparing one or more conditions with one or more other conditions.

Question

  • What effect size measure would you recommend using in relation to (a) repeated measures t-tests (b) repeated measures contrasts comparing one or more group means with one or more other group means?
  • If you were reporting a d-based effect size measure, which measure of the within group standard deviation would you use?
  • Are there any references that you would recommend?

I have a few thoughts, but I'm keen to get your suggestions.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The answer here depends on your situation. Dunlap, Cortina, Vaslow, and Burke (1996) argued that the effect size should be calculated using a SD based on pooled variance from separate conditions, as is typical in independent groups studies, even with repeated measurements. Their argument was that the study may be replicated with a between design and the effect sizes will be more comparable across studies in meta-analysis with that effect size measure. They asserted that the effect size is the effect size and shouldn't be influenced by the correlation in the measurement in a repeated measures design.

Unfortunately, this suggestion has been overgeneralized in some literatures (and in Cortina's book I believe). When it's not possible to design an experiment any other way than repeated measures then using the between S effects size is a mistake. It will underestimate the size of the effect and be useless in power calculations.

Imagine an attentional cueing study where you need to study a single mental state, (e.g. oriented in a direction indicated by an arrow), and have to measure the effect comparing performance at the indicated location and one that is not. That study has to be done within and there is no other way to do it. In that case, the need to have an effect size comparable to situations where the study is done with independent groups vanishes because the independent group study couldn't occur. Using the between S effect size would not be a useful estimate in seeing the number of S's to replicate the study while the within would. The between S effect size would tend to vastly underestimate what you actually needed to measure, which is the effect within.

Improve this answer
$\endgroup$
7
  • $\begingroup$ +1 @John I agree with Dunlap et al's arguments. I don't quite follow your claim that it has been over-generalised. In g-power 3 for example, if you want to calculate statistical power you supply the correlation between the two conditions and a between subjects measure of d. And even when a between subjects design is not possible, I would still find the between subjects measure of effect size a more useful standardisation metric for describing the amount of change that has been observed. $\endgroup$
    – Jeromy Anglim
    Commented Jun 7, 2011 at 6:47
  • $\begingroup$ When I've seen people follow the Dunlap et al recommendation I've not seen the correlation reported. Reporting the correlation would help. $\endgroup$
    – John
    Commented Jun 7, 2011 at 13:52
  • $\begingroup$ How is between S effect size more useful when only repeated measures are possible? $\endgroup$
    – John
    Commented Jun 7, 2011 at 13:59
  • $\begingroup$ @John Good point about power analysis. Having only the d without the correlation does lead to some guess work. I guess standardisation is an attempt to make a metric that lacks an intuitive or comparable interpretation have an intuitive and comparable interpretation. Thus, the sample standard deviation provides one such metric. If I want to contextualise the size of an effect, I want to be able to be able to interpret effect sizes on a common playing field. I don't want the effect size to appear larger just because it's been calculated using a repeated measures design. $\endgroup$
    – Jeromy Anglim
    Commented Jun 7, 2011 at 14:19
  • 1
    $\begingroup$ Doesn't generalized eta-square, available via ezANOVA from the ez package, solve the problem of generating an effect size estimate that generalizes across experimental designs? $\endgroup$
    – Mike Lawrence
    Commented Jun 7, 2011 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.