1
$\begingroup$

This is a cross-validation question.

I am running into a problem with a database and am using CMA. I am under the impression that once I compute effect size (d) and variance I should be able to accurately estimate sample size (N) for that comparison using the following formula, N_Calc:

N=(8+d^2)/(2*var)

I have no problems reverse engineering N when I check my between-subjects comparisons, and N_Calc is providing accurate estimates that closely match the actual N for those comparisons. So as far as cross-validation goes with those particular designs I am fine.

Where I run into problems is within-subjects designs. In many cases, of course, I don't have access to much more than N, t-values, p-values. However I do have access to some studies in which I can access raw data or access the specific comparisons - with the Pearson's r computation between means for each comparison. That's an ideal situation insofar as we know within-subjects experiments will typically show some correlation between treatment means. My problem is this: when I plug in that information into the appropriate column in CMA, I am unable to use N_Calc to get an accurate estimate of the original N.

Let's take this example:

So I have a study where the outcome is reaction time - control group (M = 1012.143, sd = 170.188) and treatment (M = 944.00, sd = 163.405). The Pearson's r between the means is .872. The actual N is 106. When I plug that info into the appropriate computation column in CMA, my d = .407 and I have a var = .003.

When I reverse engineer the N for the purpose of cross-validation, here is what happens:

N=(8+.407^2)/(2*.003) = 8.165649/.006=1360.915

Out of curiosity, I tried reversing the sign of Pearson's r. When I do that, here is what I get:

d = .409 and var = .038.

When I try to reverse engineer the N, here is what happens:

N=(8+.409^2)/(2*.038)=8.167281/.076 = 107.46

That actually comes close to approximating my actual N, which on the surface seems like what I want - i.e., demonstration that I can apparently cross-validate from d and variance and reproduce the original N with some degree of certainty.

My worry is that although I appear to have solved a problem on the surface, I am very unconfident that this is a correct solution.

My question is partially whether the N_Calc formula that I am using to reverse engineer N is applicable to situations outside of between-subjects comparisons. It seems like it should be since I am using d and variance info to plug into the formula, but I am genuinely not sure. If the N_Calc formula is applicable, is my solution to my problem with within-subjects comparisons appropriate, or am I doing something wrong? If I am doing something wrong, what do I need to do in order to get accurate computations?

Bear in mind that my only meta-analytic experiences to date have been strictly ones in which the studies were between-subjects designs. I have never dealt with within-subjects designs in a meta-analysis, nor have I tried to cross-validate effect size calculations using N_Calc formula with within-subjects designs and could use some guidance.

Thank you.

Update:

I tried one other approach to reproduce N from d and var that on the surface appears to work. I'm just following a hunch. Here it is:

I used the square root of the denominator term in the original N_Calc formula. I don't have the mathematical sophistication to justify this approach with any formal proof. It's just a hunch, and it should be treated as such for the time being:

Now, when I reverse engineer the N for the purpose of cross-validation, here is what happens:

N=(8+.407^2)/sqrt(2*.003) = 8.165649/.077459666=105.418

The advantage is that I am using the actual original correlation rather than manipulating correlation by reversing signs, my original d remains as it should as does my original variance as computed, and I appear to be cross-validating by reproducing a close approximation of the original N, which as you will recall is N=106.

Again, I have no mathematical proof to justify this approach to reproducing N from within-subjects d and variance, but on the surface, it appears to work. I reproduced N for another comparison in another report and was able to get a reasonable approximation.

So again, hoping to tap into the brainpower here to help me figure out if I am on the right track. If I am, great. If not, I'd love to know. More importantly, I'd want to know what that right track is.

Thank you for any guidance you can offer me.

$\endgroup$
1
$\begingroup$

First of all, for a between-subjects design, the (large-sample) variance of a standardized mean difference (Cohen's d) is given by $$v = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d^2}{2(n_1+n_2)}.$$ In the special case that $n_1 = n_2 = n$, we can simplify this to $$v = \frac{8 + d^2}{4n}$$ or $$v = \frac{8 + d^2}{2N},$$ where $N$ is the total sample size. Rearranging this yields $$N = \frac{8 + d^2}{2v}.$$ But note that this is a special case where the group sizes are identical.

For a within-subjects design, the (large-sample) variance of the standardized mean change is $$v = \frac{2(1-r)}{n} + \frac{d^2}{2n}.$$ This can be rearranged to $$n = \frac{2(1-r)}{v} + \frac{d^2}{2v}.$$

In your example, this yields $$\frac{2(1-.872)}{.0032} + \frac{.407^2}{2(.0032)} = 105.9.$$ I used $v = .0032$ (instead of just $v = .003$) to reduce the rounding error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.