Does it make sense to conclude that an effect is (likely) stronger in population A than in population B because the effect size estimated on A is larger than that estimated on B? (The samples are not paired. Also, the populations may have very differently shaped distributions; in particular, B may be much more peaked at 0.)

The effect size estimator I am working with looks like a variation on Cohen's $d$: $$ \frac{\mathrm{mean}(\mathcal{X}) - \mathrm{mean}(\mathcal{Y})}{\mathrm{stddev}(\mathcal{X} \cup \mathcal{Y})}, $$ where $\mathcal{X}$ and $\mathcal{Y}$ are disjoint samples from a given population. However, if the answer to my question is "no" for this estimator and "yes" for another (perhaps more common) one, I'd definitely want to know that.

My intuition is that because the effect size is normalized, it shouldn't matter how different the populations are---at least in the large-sample case. Based on the Wikipedia article, I think that Cohen's $d$ with pooled standard deviations on paired samples are directly comparable across populations, assuming those samples have the same size $N$, because $d \sqrt{N}$ is $t$-distributed. In the small-sample case I suspect it becomes less comparable because the sample mean is not as well approximated by a normal distribution. As for the specific estimator I'm using, if the samples have very different means then it seems like I'm over-normalizing---dividing the difference by an overestimate of the standard deviation---and to the extent that overestimation is different between populations A and B, the effect size estimates are not comparable. I am not sure how the lack of paired samples impacts the problem (vaguely it seems to make the effect sizes less comparable). Finally, there may be numerical error in dividing by a number potentially close to zero, but I am not asking about that issue here.

This question seems related to How to compare the effect size of two significantly different mixed liner models computed by lmer?, in which Cohen's $d$ is not applicable but it seems weakly implied that Cohen's $d$ can be compared across populations, and questions like Interpreting effect size, which attach a qualitative interpretation to different effect sizes and as a result also seem to suggest that effect sizes have meaning that generalizes beyond individual populations. Answers to this question on ResearchGate seem to indicate "yes (with caveats)." In particular, the usage of effect sizes in meta-analyses suggest that effect sizes are comparable across populations. But I'm wary of leaning too heavily on the norms of specific disciplines (whose data might look very different from mine); I don't think I've seen an answer to the effect of "yes, under these conditions, and here's why."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.