2
$\begingroup$

I am doing a univariate meta-analysis as well as TSSEM analysis, using the Theory of planned behavior, and correlation coefficients as effect sizes, but do not know what approach is best when dealing with multiple types of dependency in my data. Would you be so kind as to explain to me what options are available to me?

Out of approximately 80 studies, in 10 studies the authors conducted their research on multiple independent samples within the same study (primarily, but not exclusively, from different countries). If I am not mistaken these effect sizes are not dependent but fall into the Konstantopoulos2011 example of clustered data structures.

In 2 additional studies the authors have examined different behavior outcomes on the same sample (eg. committing music piracy, committing software piracy and committing movie piracy).

One additional study only provided correlation coefficients and sample sizes with regards to men and women separately, but did not report effect sizes for the whole sample.

Therefore I have three sources of effect size dependency but for a small number of studies.

I am (in theory) familiar that for these cases one may utilize: averaging, three-level meta analysis, robust variance estimation, correlated and hierarchical effects (CHE). I obtained several articles and r-scripts from OSF but it seems to me that most only deal with individual causes of data dependency.

May I use three-level meta-analysis even-though I do not account for the dependency of sampling errors of the observed effects within several studies that reported multiple outcomes based on the same sample, having in mind that 90% of the dependency in the studies follows a clustered/hierarchical structure?

Can you please give me advice?

Thank you in advance.

$\endgroup$

1 Answer 1

4
$\begingroup$

Yes, if a study provides you with multiple correlation coefficients for different subgroups (e.g., men versus women) or different independent samples, then this leads to a multilevel structure that you can account for using a multilevel random-effects model as described by Konstantopoulos (2011). In essence, the issue here is that the true correlations corresponding to the multiple observed correlations within a study might be correlated and we should account for this. Since you are dealing with correlation coefficients, you might also appreciate the illustrative analysis of the data by Credé et al. (2010) (on the relationship between class attendance and grades). So the first and the third sources of dependency you mention are in essence of the same type and can be handled in the same manner.

The second type you mention is more difficult to deal with. For studies that provide correlations between X and Y for different measures of Y (say you have $\mbox{cor}(X,Y_1)$ and $\mbox{cor}(X,Y_2)$), then this leads to the additional complexity that the sampling errors of these two correlation coefficients are not independent. If you know (or can make a reasonable guess about) $\mbox{cor}(Y_1,Y_2)$, then one can compute the covariance between the two correlations coefficients (or their r-to-z transformed versions), for example with the rcalc() function from the metafor package. In essence then, you need to create the entire variance-covariance matrix for the correlation coefficients (the $V$ matrix) and for studies of this type, the off-diagonal elements will not be zero. Beyond this, you also have again multiple estimates coming from the same study whose underlying true correlations might be correlated, but this we are already dealing with via the multilevel model.

To give one last example, you might also want to take a look at the meta-analysis by Knapp et al. (2017) on differences in planning performance in schizophrenia patients versus healthy controls. This is an example where we have this mix of different dependencies. In fact, this example is even a bit more complex, since correlations between the sampling errors of the effect sizes (standardized mean differences in this example) could arise due to different reasons. This led to the admittedly convoluted code for constructing the $V$ matrix, but I would not get too hung up about this, since this issue might not apply in your case. Instead, you probably need rcalc(), which also requires some time to get familiar with and you need to structure your data accordingly.

Coming back to the example above, your dataset then need to be of this form:

study sample var1 var2 r
------------------------
1     1      X    Y    .
2     1      X    Y    .
2     2      X    Y
3     1      X    Y1   .
3     1      X    Y2   .
3     1      Y1   Y2   .

So study 1 just provides a single correlation, study 2 has two independent samples, and study 3 is like the one described above. For the purposes of rcalc(), a 'study' is defined as an independent sample, so you need to combine 'study' and 'sample' above into an identifier (e.g., paste0(dat$study, ".", dat$sample)), so that samples 1 and 2 from study 2 will be treated as independent (which they are). Then you can use rcalc() as described in the documentation.

In any case, I would recommend to also do a sensitivity analysis using cluster-robust inference methods as shown in the Knapp example. Ssee also the recommendations given here.

Addendum (based on the comments):

One can do a 3-level model even if only some studies provide multiple estimates. First, consider those 5 studies with 2 samples, which provide 10 estimates in total. Given potential dependency (at the higher level), the effective sample size (effective number of estimates) is somewhere between 5 and 10, probably somewhere in-between, and few would object to fitting a standard RE model to ~7 estimates. So if one is willing to fit standard RE models to 7 or so independent estimates, then one should have no qualms about estimating the variance component at the estimate level in the three-level model. And at the higher level, there are way more units (studies), so that's no problem either.

For cluster-robust inference methods (also known as robust variance estimation), there are small-sample corrections that one should use and estimate the degrees of freedom using a Satterthwaite approximation (this is all available via robust(..., clubSandwich=TRUE) in the metafor package, making use of the methods in the clubSandwich package). If the degrees of freedom are really low, then one could also consider cluster wild bootstrapping, which is implemented in the wildmeta package (see the notes here).

As for averaging the two dependent estimates - doing this right requires knowing something about the correlation between the two estimates in the first place. Simply taking their average and pretending it was one single correlation is not correct. In essence, this implicitly assumes that $\mbox{cor}(Y_1,Y_2) = 1$). But what you do here will probably have very little influence on the results, given that this is just two pairs of estimates. So using a very rough guess about $\mbox{cor}(Y_1,Y_2)$ is fine, also when using cluster-robust inference methods in the end.

$\endgroup$
7
  • $\begingroup$ Dear Wolfgang, thank you very much. Can you please clarify can someone really do a 3-level/CHE/RVE when only several of the studies exhibit dependencies (only 5/80 have 2 independent samples each; only 2/80 use the same sample and basically the same questionnaire only varying the type of piracy behavior)? Isn't there too little studies with dependent ES for the (heterogeneity) estimates to be meaningful? Several articles seem to suggest that one would for example need much more dependent ES per study and in much more studies, to consider some of these approaches? Kind regards, Miljan $\endgroup$
    – Miljan
    Commented Sep 12 at 20:27
  • $\begingroup$ Dear Wolfgang, additionally, it is still not clear to me (my fault; I am new to meta-analysis), how can I take into account of all three types of dependencies all in one model? It seems to me that you suggest the multivariate approach as the right one, but you are right - I do not know/have a reasonable guess about the correlation of my dependent ES, and am perhaps insufficiently experienced to try this approach. Are there reasonable alternatives, e.g. CHE? Is averaging the 2 studies that use the same sample and then performing 3-level analysis a crude but possible route? Thanks again. $\endgroup$
    – Miljan
    Commented Sep 12 at 20:32
  • $\begingroup$ Dear Wolfgang, one last thing i promise. It is not clear to me based on the rcalc tutorial, how should I structure var1 and var2. Namely, my studies that use the same sample basically only vary the "topic" of interest but use the same variables. Therefore, my ES pairs are for example: Questionnaire1: attitude (towards music piracy) - intention (towards music piracy); Quest2: atittude (towards video piracy) - intention (towards video piracy), Quest3: ... . It seems to me that this does not fall either into the partially ‘overlapping’ case or the ‘non-overlapping’ case, or am I mistaken? Thanks $\endgroup$
    – Miljan
    Commented Sep 13 at 8:36
  • $\begingroup$ First, I just added an addendum above based on your first two comments (which would have been way too long for answering via the comments). As for your third comment: Then you actually have 4 variables (attitude music piracy, intention music piracy, attitude video piracy, intention video piracy) and you are interested in cor(attitude music piracy, intention music piracy) and cor(attitude video piracy, intention video piracy), so this is the non-overlapping case. $\endgroup$
    – Wolfgang
    Commented Sep 13 at 9:28
  • $\begingroup$ Dear Wolfgang, If I am understanding correctly, besides having ATTmp-INTmp and ATTvp-INTvp, in our example I would then need to add rows in my dataset for ATTmp-ATTvp, ATTmp-INTvp, ATTvp-INTmp and INTmp-INTvp, and then make a reasonable guess about their correlations, and then make use of the rcalc function after that? Does that sound right? $\endgroup$
    – Miljan
    Commented Sep 13 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.