Yes, if a study provides you with multiple correlation coefficients for different subgroups (e.g., men versus women) or different independent samples, then this leads to a multilevel structure that you can account for using a multilevel random-effects model as described by Konstantopoulos (2011). In essence, the issue here is that the true correlations corresponding to the multiple observed correlations within a study might be correlated and we should account for this. Since you are dealing with correlation coefficients, you might also appreciate the illustrative analysis of the data by Credé et al. (2010) (on the relationship between class attendance and grades). So the first and the third sources of dependency you mention are in essence of the same type and can be handled in the same manner.
The second type you mention is more difficult to deal with. For studies that provide correlations between X and Y for different measures of Y (say you have $\mbox{cor}(X,Y_1)$ and $\mbox{cor}(X,Y_2)$), then this leads to the additional complexity that the sampling errors of these two correlation coefficients are not independent. If you know (or can make a reasonable guess about) $\mbox{cor}(Y_1,Y_2)$, then one can compute the covariance between the two correlations coefficients (or their r-to-z transformed versions), for example with the rcalc()
function from the metafor package. In essence then, you need to create the entire variance-covariance matrix for the correlation coefficients (the $V$ matrix) and for studies of this type, the off-diagonal elements will not be zero. Beyond this, you also have again multiple estimates coming from the same study whose underlying true correlations might be correlated, but this we are already dealing with via the multilevel model.
To give one last example, you might also want to take a look at the meta-analysis by Knapp et al. (2017) on differences in planning performance in schizophrenia patients versus healthy controls. This is an example where we have this mix of different dependencies. In fact, this example is even a bit more complex, since correlations between the sampling errors of the effect sizes (standardized mean differences in this example) could arise due to different reasons. This led to the admittedly convoluted code for constructing the $V$ matrix, but I would not get too hung up about this, since this issue might not apply in your case. Instead, you probably need rcalc()
, which also requires some time to get familiar with and you need to structure your data accordingly.
Coming back to the example above, your dataset then need to be of this form:
study sample var1 var2 r
------------------------
1 1 X Y .
2 1 X Y .
2 2 X Y
3 1 X Y1 .
3 1 X Y2 .
3 1 Y1 Y2 .
So study 1 just provides a single correlation, study 2 has two independent samples, and study 3 is like the one described above. For the purposes of rcalc()
, a 'study' is defined as an independent sample, so you need to combine 'study' and 'sample' above into an identifier (e.g., paste0(dat$study, ".", dat$sample)
), so that samples 1 and 2 from study 2 will be treated as independent (which they are). Then you can use rcalc()
as described in the documentation.
In any case, I would recommend to also do a sensitivity analysis using cluster-robust inference methods as shown in the Knapp example. Ssee also the recommendations given here.
Addendum (based on the comments):
One can do a 3-level model even if only some studies provide multiple estimates. First, consider those 5 studies with 2 samples, which provide 10 estimates in total. Given potential dependency (at the higher level), the effective sample size (effective number of estimates) is somewhere between 5 and 10, probably somewhere in-between, and few would object to fitting a standard RE model to ~7 estimates. So if one is willing to fit standard RE models to 7 or so independent estimates, then one should have no qualms about estimating the variance component at the estimate level in the three-level model. And at the higher level, there are way more units (studies), so that's no problem either.
For cluster-robust inference methods (also known as robust variance estimation), there are small-sample corrections that one should use and estimate the degrees of freedom using a Satterthwaite approximation (this is all available via robust(..., clubSandwich=TRUE)
in the metafor package, making use of the methods in the clubSandwich
package). If the degrees of freedom are really low, then one could also consider cluster wild bootstrapping, which is implemented in the wildmeta
package (see the notes here).
As for averaging the two dependent estimates - doing this right requires knowing something about the correlation between the two estimates in the first place. Simply taking their average and pretending it was one single correlation is not correct. In essence, this implicitly assumes that $\mbox{cor}(Y_1,Y_2) = 1$). But what you do here will probably have very little influence on the results, given that this is just two pairs of estimates. So using a very rough guess about $\mbox{cor}(Y_1,Y_2)$ is fine, also when using cluster-robust inference methods in the end.