3
$\begingroup$

I'm conducting a meta-analysis using the metafor package. I'm using partial correlations and I'm interested in two effects - correlations between X and Y1 and X and Y2 (both controlling for Z).

I have overall estimates for both effects now and want to test if one effect is bigger than the other (all the individual studies I'm using have both outcomes so they are based on the same datasets).

Can anyone give me an advice on how I should precede? In short, I have two meta analytic estimates (based on the same individual datasets) like below and want to compare them:

estimate1 (partial cor) = 0.30, SE = 0.05, z = 6.62

estimate2 (partial cor) = 0.15, SE = 0.02, z = 7.30

Any help would be very much appreciated!

$\endgroup$

1 Answer 1

3
$\begingroup$

Since the partial correlations of $X$ and $Y_1$ (the $r_{X,Y_1}$ values) and the partial correlations of $X$ and $Y_2$ (the $r_{X,Y_2}$ values) come from the same set of studies, they are not independent. Therefore, the pooled estimate of the $r_{X,Y_1}$ values (say $\bar{r}_{X,Y_1}$) is also not independent of the pooled estimate of the $r_{X,Y_2}$ values (say $\bar{r}_{X,Y_2}$). Therefore, to properly test if the difference between the two pooled estimates is significant, we must know something about the degree of dependence between the two pooled estimates. If we would have this information, we could use a Wald-type test whose test statistic is given by $$z = \frac{\bar{r}_{X,Y_1} - \bar{r}_{X,Y_2}}{\sqrt{\mbox{SE}[\bar{r}_{X,Y_1}]^2 + \mbox{SE}[\bar{r}_{X,Y_2}]^2 - 2 \times r_{\bar{r}_{X,Y_1},\bar{r}_{X,Y_2}} \times \mbox{SE}[\bar{r}_{X,Y_1}]\mbox{SE}[\bar{r}_{X,Y_2}]}},$$ where $r_{\bar{r}_{X,Y_1},\bar{r}_{X,Y_2}}$ denotes the correlation between the two pooled estimates. Since $r$ is not known here, you could examine what happens for all possible values of $r_{\bar{r}_{X,Y_1},\bar{r}_{X,Y_2}}$ between $-1$ and $+1$. Here is a plot of the z-value from this test as a function of $r_{\bar{r}_{X,Y_1},\bar{r}_{X,Y_2}}$:

value of z as a function of r

As you can see, the z-value is always larger than $1.96$, the upper critical value for a z-test, and hence, no matter what the value of $r_{\bar{r}_{X,Y_1},\bar{r}_{X,Y_2}}$ actually is, the test would be significant. So you can be sure that the difference is significant - you just can't give the exact p-value, but you can say $p < .05$. Or actually, since the smallest z-value is $2.143$, this implies that the largest possible (two-sided) p-value is $p_{max} = .032$, so you can even say $p \le .032$.

$\endgroup$
2
  • $\begingroup$ THANK YOU SO MUCH! This makes a lot of sense. 1) Is there any readings on this that you could point to? 2) I was also wondering if the CI for the two effects don't overlap, if it's possible to infer that they are significantly different? I really appreciate your help with this!! $\endgroup$
    – llbia
    Commented Sep 22, 2021 at 13:10
  • $\begingroup$ 1) This is based on elementary statistical theory, so it's difficult to give a reference that describes just this. 2) Yes, if the (Wald-type) CIs do not overlap, then you know that the Wald-type test given above would be significant even in the most extreme scenario where $r = -1$. Note that the reverse does not hold in general (i.e., even if the two CIs overlap, then the test of the difference may still be significant). $\endgroup$
    – Wolfgang
    Commented Sep 22, 2021 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.