2
$\begingroup$

I'm trying to perform a meta-analysis where my effects are mainly one-sample t-test against chance level. I've easily found the one-sample cohen's d formula but I'm not able to find the variance formula to weight the estimated effect size. I've found this post however I would like to have a clear reference or maybe other suggestions. Thanks!

$\endgroup$
2
  • $\begingroup$ Could you elaborate on what this variance would be of? If it's the estimated effect size in a given experiment, then that's precisely what appears in the numerator of the Student $t$ statistic and an estimate of its variance (adjusted by a factor of $1/\sqrt{n}$) appears in the denominator. $\endgroup$
    – whuber
    Jun 23, 2020 at 12:20
  • $\begingroup$ I would like to compute the variance of the effect-size to perform the inverse-variance weighting for the meta-analysis. For example from the book of Borenstein (2009) for each effect size measure (not the one-sample cohen's d) there is the formula for the variance also. $\endgroup$ Jun 23, 2020 at 12:53

1 Answer 1

1
$\begingroup$

First off, it is important to clarify the nature of your effect size. There are two ways to standardized the mean difference. The first is with either the time 1 standard deviation or the pooled time 1 and time 1standard deviations. The second is with the standard deviation of the gain scores. The relationship between these two denominators is $$ s = \frac{s_{gain}}{\sqrt{2 \left( 1 - r \right)}} $$ where $s$ is the pooled or raw standard deviation and $r$ is the correlation between the time 1 and time 2 scores. I strongly recommend standardizing on $s$ and not $s_{gain}$. It is not clear how to interpret the latter as you are standardizing the mean change on the variability in change.

Assuming you are standardizing the mean difference using $s$, then the variance for the effect size is $$ v = \frac{2*(1-r)}{n} + \frac{d^2}{2*n} ~.$$ The details for this are worked out in:

Becker, B. J. (1988). Synthesizing standardized mean‐change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278.

The challenge, of course, is that in the context of meta-analysis, the studies often do not report $r$. Some meta-analysts use a guesstimate of $r = .5$ and you can do sensitivity analysis with different values. If the study reports the time 1 and time 2 standard deviations and means and one-sample t-test, then you can recover $r$ using the following: $$ r = \frac {\left(s_1^2 t^2 + s_2^2 t^2 \right) - \left(\overline{x}_2 - \overline{x}_1\right) n} {2 s_1 s_2 t^2} ~.$$ This assumes that the sign of $t$ matches the sign of the mean difference. Flip one or both as needed.

$\endgroup$
3
  • $\begingroup$ Thanks for your detailed explanation and for the reference. However, I think that I'm missing something. What are Time 1 and Time 2? In my experiments the performance, let's say 58% (SD=0.03) accuracy is compared to the chance level. So I can compute the cohen's d $d=\frac{\bar x - chance\ level}{s}$. I don't have two measures for the subject. $\endgroup$ Jun 25, 2020 at 16:54
  • $\begingroup$ I had assumes two repeated measures or paired data, as is typical of a dependent t-test. I misread your post. Sorry. You have a one-sample t-test with an non-zero null. The inverse variance is simply 1/n (the standard error of your d is simply 1/sqrt(n). $\endgroup$
    – dbwilson
    Jun 29, 2020 at 12:32
  • $\begingroup$ thanks for your time! so the formula inside this post is not correct right? $\endgroup$ Jun 29, 2020 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.