1
$\begingroup$

I want to conduct a comparative analysis across a large amount of studies by looking at the effect sizes of their results with another variable. I am not too familiar with effect sizes so I'd appreciate some help.

All the data is non-parametric and some studies use different tests (Wilcoxon rank sum for the most part but sometimes ANOVAs or Mann-Whitney U tests). Is there a meaningful way to compare the effect sizes across all these studies? About half of the studies present the data with means and standard errors/deviations from which I can get Cohen's d, but is this applicable to non-normal data? Or can I just use the r=Z/SQRT(N) formula on all the test values? Additionally, can I compare effect sizes obtained from means + standard deviations with effect sizes obtained from test statistics?

Any help would be appreciated, clearly I'm a bit lost.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ How have others in your field dealt with this issue and what kind of non-normality are you talking about? Or better yet, what kind of data are you talking about? Please add those to the question. $\endgroup$
    – John
    Commented Sep 11, 2016 at 19:40

1 Answer 1

Reset to default
1
$\begingroup$

Cohen's D is not a reliable effect size for non-parametric methods.

I have seen the formula r=Z/SQRT(N) suggested here on this forum but am unsure about its assumptions. In my reading about non-parametric effect sizes I have learnt that factors such as skewed distributions, random sampling or statistical independence can influence the reliability of a non-parametric effect size measure. source 1, source 2, source 3, source 4.

I don't see any problem if you use means and standard deviations to compare effect sizes based on parametric tests. But I would be careful applying these measures to non-parametric test results.

Comparing Cohen's D formula with the non-parametric one you select (there are many different ones) gives you an idea how comparable the effect sizes are.

I guess the crux is that you need more information about the samples of the studies you are comparing, which you probably don't have. How have other meta reviews in your field dealt with it?

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I'd be a bit cautious about some of those readings. I didn't go through them all but there are some notable errors / misconceptions. The American Psychologist article is replete with a focus on meeting test assumptions for particular data rather than the population. The few areas where populations could be interpreted they make errors. For example, it is true that reaction times (RTs) are skewed but they are generally aggregated before being analyzed with ANOVA and mean RTs (with a decent number of measures per condition) are often not skewed. $\endgroup$
    – John
    Commented Sep 11, 2016 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.