Estimating N from d and variance for within-studies designs in meta-analysis

Question

This is a cross-validation question.

I am running into a problem with a database and am using CMA. I am under the impression that once I compute effect size (d) and variance I should be able to accurately estimate sample size (N) for that comparison using the following formula, N_Calc:

N=(8+d^2)/(2*var)

I have no problems reverse engineering N when I check my between-subjects comparisons, and N_Calc is providing accurate estimates that closely match the actual N for those comparisons. So as far as cross-validation goes with those particular designs I am fine.

Where I run into problems is within-subjects designs. In many cases, of course, I don't have access to much more than N, t-values, p-values. However I do have access to some studies in which I can access raw data or access the specific comparisons - with the Pearson's r computation between means for each comparison. That's an ideal situation insofar as we know within-subjects experiments will typically show some correlation between treatment means. My problem is this: when I plug in that information into the appropriate column in CMA, I am unable to use N_Calc to get an accurate estimate of the original N.

Let's take this example:

So I have a study where the outcome is reaction time - control group (M = 1012.143, sd = 170.188) and treatment (M = 944.00, sd = 163.405). The Pearson's r between the means is .872. The actual N is 106. When I plug that info into the appropriate computation column in CMA, my d = .407 and I have a var = .003.

When I reverse engineer the N for the purpose of cross-validation, here is what happens:

N=(8+.407^2)/(2*.003) = 8.165649/.006=1360.915

Out of curiosity, I tried reversing the sign of Pearson's r. When I do that, here is what I get:

d = .409 and var = .038.

When I try to reverse engineer the N, here is what happens:

N=(8+.409^2)/(2*.038)=8.167281/.076 = 107.46

That actually comes close to approximating my actual N, which on the surface seems like what I want - i.e., demonstration that I can apparently cross-validate from d and variance and reproduce the original N with some degree of certainty.

My worry is that although I appear to have solved a problem on the surface, I am very unconfident that this is a correct solution.

My question is partially whether the N_Calc formula that I am using to reverse engineer N is applicable to situations outside of between-subjects comparisons. It seems like it should be since I am using d and variance info to plug into the formula, but I am genuinely not sure. If the N_Calc formula is applicable, is my solution to my problem with within-subjects comparisons appropriate, or am I doing something wrong? If I am doing something wrong, what do I need to do in order to get accurate computations?

Bear in mind that my only meta-analytic experiences to date have been strictly ones in which the studies were between-subjects designs. I have never dealt with within-subjects designs in a meta-analysis, nor have I tried to cross-validate effect size calculations using N_Calc formula with within-subjects designs and could use some guidance.

Thank you.

Update:

I tried one other approach to reproduce N from d and var that on the surface appears to work. I'm just following a hunch. Here it is:

I used the square root of the denominator term in the original N_Calc formula. I don't have the mathematical sophistication to justify this approach with any formal proof. It's just a hunch, and it should be treated as such for the time being:

Now, when I reverse engineer the N for the purpose of cross-validation, here is what happens:

N=(8+.407^2)/sqrt(2*.003) = 8.165649/.077459666=105.418

The advantage is that I am using the actual original correlation rather than manipulating correlation by reversing signs, my original d remains as it should as does my original variance as computed, and I appear to be cross-validating by reproducing a close approximation of the original N, which as you will recall is N=106.

Again, I have no mathematical proof to justify this approach to reproducing N from within-subjects d and variance, but on the surface, it appears to work. I reproduced N for another comparison in another report and was able to get a reasonable approximation.

So again, hoping to tap into the brainpower here to help me figure out if I am on the right track. If I am, great. If not, I'd love to know. More importantly, I'd want to know what that right track is.

Thank you for any guidance you can offer me.

Answer 1

First of all, for a between-subjects design, the (large-sample) variance of a standardized mean difference (Cohen's d) is given by $$v = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d^2}{2(n_1+n_2)}.$$ In the special case that $n_1 = n_2 = n$, we can simplify this to $$v = \frac{8 + d^2}{4n}$$ or $$v = \frac{8 + d^2}{2N},$$ where $N$ is the total sample size. Rearranging this yields $$N = \frac{8 + d^2}{2v}.$$ But note that this is a special case where the group sizes are identical.

For a within-subjects design, the (large-sample) variance of the standardized mean change is $$v = \frac{2(1-r)}{n} + \frac{d^2}{2n}.$$ This can be rearranged to $$n = \frac{2(1-r)}{v} + \frac{d^2}{2v}.$$

In your example, this yields $$\frac{2(1-.872)}{.0032} + \frac{.407^2}{2(.0032)} = 105.9.$$ I used $v = .0032$ (instead of just $v = .003$) to reduce the rounding error.