3
$\begingroup$

I submitted a paper in which I analyzed the data by a mixed-design ANOVA. I reported generalized eta squared as effect size. After significant interactions, I conducted planned comparisons. A reviewer suggested that I have to report planned comparisons only when an effect is significant with a sufficiently large effect size, based on Cohen's rules of thumb.

My question is: what is to be considered as a sufficiently large effect size? I'm not sure of what is (for the reviewer) a sufficiently large effect size. Following Cohen's rules of thumb, I would consider a generalized eta squared of at least .02 (small) as sufficient to conduct planned comparisons. In your opinion, is this approach correct or do I have to consider only significant effects with medium to large effect sizes?

Thanks.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Maybe you'll find this helpful: imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/effectSize $\endgroup$
    – Tim
    Commented Dec 4, 2014 at 9:11
  • $\begingroup$ Thank you Tim. I had already read that, but my question regards the meaning of "sufficiently large". In your opinion, may a generalized eta squared of .02 be considered sufficiently large for planned comparisons? $\endgroup$
    – this.is.not.a.nick
    Commented Dec 4, 2014 at 9:34
  • $\begingroup$ "Sufficiently" is not really precise, so you could guess if your reviewer meant rather "medium" or "large". It is up to you. You can always answer your reviewer why while your effect is of certain size your results have a practical significance that makes them valuable (if it is so). $\endgroup$
    – Tim
    Commented Dec 4, 2014 at 9:39
  • $\begingroup$ I also think that "sufficiently" is a bit imprecise For this reason I posted this question. Thanks again. $\endgroup$
    – this.is.not.a.nick
    Commented Dec 4, 2014 at 9:42

2 Answers 2

Reset to default
2
$\begingroup$

Jacob Cohen established thresholds for some standardized effect sizes, to which he attached the adjectives small/medium/large. However, to the best of my knowledge, he never suggested that any of these values should be considered as "sufficient" in general. In fact, in his book about power analysis (1988), he advised to avoid as much as possible using these thresholds, and repeatedly said that context should always take precedence to judge the practical relevance of an effect size.

In your case, you may want to present a rationale to the reviewer as to why you consider the effect size as important (or not) in the context of your study.

For example, consider a study about an intervention meant to increase people's income. After conducting the study, we observe that the intervention increases people's income by \$50 per month on average, compared to the control group, who experiences no increase. Is this effect size of +50 "sufficiently large"? It depends on the cost of the intervention, on the original income of these people, on whether alternative interventions in other studies lead to a larger or smaller increase, possibly on how this average increase is distributed among people, etc.

In short, you have to relate this effect size to the goal of your study, to say if it is "sufficiently large" or not.

If you have really no idea if a given effect size is relevant or not because the context of your study does not lend itself to this kind of considerations, I think that using adjectives like "small", "medium", or "large" in a way completely decoupled from context is more confusing, than simply reporting the value of the effect size.

However, if the reviewer expect from you to mention Cohen's thresholds, then you could say something along the lines of "This effect size of 0.02 is in line with the definition of 'small' as defined by Cohen (1988)" and add a clear caveat as to the arbitrariness of these thresholds, to incite readers to judge the situation for themselves (you can find some references about that here, including some quotes by Cohen).

If you're in the case of planning a future study and struggle with defining an effect size of interest in order to compute a required sample size, note that this paper by Daniel Lakens (2022) provides some useful guidance on how to justify sample size for a variety of situations.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). L. Erlbaum Associates.

Lakens, D. (2022). Sample Size Justification. Collabra: Psychology, 8(1), 33267. https://doi.org/10.1525/collabra.33267

Improve this answer
$\endgroup$
2
$\begingroup$

The answer by triple J here is already sufficient (+1). I would like to add perhaps another informative answer.

The history behind effect sizes is interesting and has been going on for longer than is given credit. Psychologists often know effect sizes via Cohen as suggested in the other answer, but these precluded him a long time ago (Huberty, 2002). Since that time, researchers and statisticians have tried to provide informative advice about how to use effect sizes in a useful way, but best efforts have been turned into a haphazard methods. Nowhere is this more clear than the cutoff criterions made by Cohen, which were at the time well-intentioned and still useful, but nonetheless don't serve the purposes of helping researchers in every situation in modern contexts.

Let us use mixed models as an example. For a long time there was no such thing as effect sizes (at least in the conventional sense) for mixed models. It was only until Nakagawa and Schielzeth published a couple papers in 2013 that this came to the forefront (see references below). But this only addressed one aspect, which was model effect size. They later addressed this in a later paper which defined part $R^2$ (Stoffel et al., 2021). And yet because partial effect sizes are still so new (I rarely see them reported in my area of research), it is very unclear what can be considered small or large.

Now what does that mean for your specific scenario? We often have to ask ourselves what is considered a useful effect size in your situation. Is $\eta^2 = .02$ a common effect size in your field? If so, what would be considered a value which you would consider extreme? This would do a great job of informing you and others of what is useful in this context. I would use past research as some barometer of which effect sizes are useful. Thankfully, $\eta^2$ and partial $\eta^2$ have been around for some time, so you should be able to find a number of articles that report this, and hopefully inform you of what is useful.

References

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. https://doi.org/10.1177/2515245919847202

Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227–240. https://doi.org/10.1177/0013164402062002002

Nakagawa, S., & Schielzeth, H. (2013a). A general and simple method for obtaining R 2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012.00261.x

Nakagawa, S., & Schielzeth, H. (2013b). A general and simple method for obtaining R 2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012.00261.x

Stoffel, M. A., Nakagawa, S., & Schielzeth, H. (2021). partR2: Partitioning R2 in generalized linear mixed models. PeerJ, 9, e11414. https://doi.org/10.7717/peerj.11414

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.