This is a cross-validation question.
I am running into a problem with a database and am using CMA. I am under the impression that once I compute effect size (d) and variance I should be able to accurately estimate sample size (N) for that comparison using the following formula, N_Calc:
I have no problems reverse engineering N when I check my between-subjects comparisons, and N_Calc is providing accurate estimates that closely match the actual N for those comparisons. So as far as cross-validation goes with those particular designs I am fine.
Where I run into problems is within-subjects designs. In many cases, of course, I don't have access to much more than N, t-values, p-values. However I do have access to some studies in which I can access raw data or access the specific comparisons - with the Pearson's r computation between means for each comparison. That's an ideal situation insofar as we know within-subjects experiments will typically show some correlation between treatment means. My problem is this: when I plug in that information into the appropriate column in CMA, I am unable to use N_Calc to get an accurate estimate of the original N.
Let's take this example:
So I have a study where the outcome is reaction time - control group (M = 1012.143, sd = 170.188) and treatment (M = 944.00, sd = 163.405). The Pearson's r between the means is .872. The actual N is 106. When I plug that info into the appropriate computation column in CMA, my d = .407 and I have a var = .003.
When I reverse engineer the N for the purpose of cross-validation, here is what happens:
N=(8+.407^2)/(2*.003) = 8.165649/.006=1360.915
Out of curiosity, I tried reversing the sign of Pearson's r. When I do that, here is what I get:
d = .409 and var = .038.
When I try to reverse engineer the N, here is what happens:
N=(8+.409^2)/(2*.038)=8.167281/.076 = 107.46
That actually comes close to approximating my actual N, which on the surface seems like what I want - i.e., demonstration that I can apparently cross-validate from d and variance and reproduce the original N with some degree of certainty.
My worry is that although I appear to have solved a problem on the surface, I am very unconfident that this is a correct solution.
My question is partially whether the N_Calc formula that I am using to reverse engineer N is applicable to situations outside of between-subjects comparisons. It seems like it should be since I am using d and variance info to plug into the formula, but I am genuinely not sure. If the N_Calc formula is applicable, is my solution to my problem with within-subjects comparisons appropriate, or am I doing something wrong? If I am doing something wrong, what do I need to do in order to get accurate computations?
Bear in mind that my only meta-analytic experiences to date have been strictly ones in which the studies were between-subjects designs. I have never dealt with within-subjects designs in a meta-analysis, nor have I tried to cross-validate effect size calculations using N_Calc formula with within-subjects designs and could use some guidance.
I tried one other approach to reproduce N from d and var that on the surface appears to work. I'm just following a hunch. Here it is:
I used the square root of the denominator term in the original N_Calc formula. I don't have the mathematical sophistication to justify this approach with any formal proof. It's just a hunch, and it should be treated as such for the time being:
Now, when I reverse engineer the N for the purpose of cross-validation, here is what happens:
N=(8+.407^2)/sqrt(2*.003) = 8.165649/.077459666=105.418
The advantage is that I am using the actual original correlation rather than manipulating correlation by reversing signs, my original d remains as it should as does my original variance as computed, and I appear to be cross-validating by reproducing a close approximation of the original N, which as you will recall is N=106.
Again, I have no mathematical proof to justify this approach to reproducing N from within-subjects d and variance, but on the surface, it appears to work. I reproduced N for another comparison in another report and was able to get a reasonable approximation.
So again, hoping to tap into the brainpower here to help me figure out if I am on the right track. If I am, great. If not, I'd love to know. More importantly, I'd want to know what that right track is.
Thank you for any guidance you can offer me.