1
$\begingroup$

I am currently planning a reliability meta-analysis or reliability generalization study (Vacha-Haase, 1998). I write this question assuming a random-effects model has been selected and Cronbach's a is the reliability estimate.

There are several considerations that need to be made, including:

  1. Whether to transform the reliability coefficients (e.g., using Fisher z, Hakstian-Whalen, or Bonett transformations) or the use raw estimates.

  2. Whether to use the inverse variance or sample size as weights.

  3. Which heterogeneity estimator to use (e.g., Hunter-Schmidt, DerSimonian-Laird, REML, etc.).

Traditionally, meta-analysts have used a single method (e.g., the Hunter-Schmidt method involves raw estimates, sample-size weights, and the Hunter-Schmidt estimator of heterogeneity vs. the Hedges-Vevea method, which involves z transformed estimates for correlation coefficients [and Hakstian-Whallen/Bonett transformed estimates for Cronbach's alpha] and inverse variance weights).

This is certainly how some statistical packages operate too (e.g., with the metafor package in R, you cannot request z transformed estimates ("ZCOR") and use the sample-size weighted variance (vtype = "AV"); the packages overrides this selection with the default settings for "ZCOR".

Update: This is not completely accurate - the sampling variance is dependent on the treatment of the distribution, and in this case, it does not make any difference whether the large scale or sample-size weighted variance is used, see Wolfang's answer below. metafor does not work 'within' methods and is as flexible as other packages.

There may be certain instances where meta-analysts wish to combine aspects from different methods because certain statistical procedures suit their dataset (e.g., normality of their coefficient distribution, the range of sample sizes, etc.), they wish to run sensitivity analyses switching certain procedures, or they wish to compare different procedures in simulation studies.

My question is: is it statistically sound to combine statistical procedures from different meta-analytic methods? For example, if one were to transform their reliability coefficients, would they also need to calculate the sampling variance using the assumed procedures (e.g., inverse variance in the case of z transform vs. sample-size weighted variance, for instance)?

Update: Again, the example given here is based on the assumption that the treatment of the coefficient distribution and sampling variance estimation method are two separate things, which is not quite the case. A more appropriate question is whether different statistical procedures can be used for each decision numbered above, e.g., transformed coefficients + sample-size weights (vs. inverse variance)

I am not asking if certain methods or statistical procedures are better or worse than others (a discussion of this can be found here). Instead, I wish to know it if is statistically sound to combine aspects from different methods?

$\endgroup$

1 Answer 1

3
$\begingroup$

I don't understand how metafor limits you. vtype = "AV" has nothing to do with whether sample-size weights or inverse-variance weights are used when combining multiple raw or r-to-z transformed correlation coefficients. The default in rma() is to use inverse-variance weights, but the weights argument allows you to specify any other weights you like.

The vtype argument allows you to choose, for certain outcome measures, among different ways of calculating the sampling variances. For example, vtype = "AV" can be useful for raw correlation coefficients, since the estimates of the sampling variances can be quite inaccurate for raw correlations. For r-to-z transformed correlations, this is useless, since the r-to-z transformation is a variance-stabilizing transformation for correlation coefficients, so the sampling variances of the transformed values are exactly known.[1]

All this aside, of course you can combine different aspects of different procedures. For example, using escalc(), you can work with raw Cronbach's alpha values, or use the Hakstian-Whalen or the Bonett transformation,[2] you can use any weights you like in rma(), and you can choose among all those heterogeneity estimators you mentioned (and others).

[1] Well, not exactly, since $1/(n-3)$ is still an approximation, but it is very accurate unless $n$ is really small.

[2] escalc() does not give you the option to use the r-to-z transformation for Cronbach's alpha values, since this transformation is meant for Pearson product-moment correlation coefficients, not Cronbach's alpha values.

$\endgroup$
4
  • $\begingroup$ The meta-analysis gods have heeded my call! Thank you for clarifying these details. I mistakenly conflated the calculation of the sampling variance with the sampling weights. Based on your answer, my understanding is that they are related insofar as the use of inverse variance weights will depend on the method for calculating the sampling variance and hence the treatment of the coefficient distribution. $\endgroup$ Aug 13, 2021 at 12:16
  • $\begingroup$ I also wrongly assumed that the metafor is inflexible as I could not adjust the method for estimating the sampling variance, but this of course overlooks how the sampling variances are somewhat dependent on the treatment of the coefficient distribution (hence why changing vtype from “AV” to “LV” resulted in no difference after transforming the distribution; though it is possible to select between “AV”, “LS” and “UB” when using raw coefficients). I have updated my question to avoid misinforming others who read it. $\endgroup$ Aug 13, 2021 at 12:16
  • $\begingroup$ As for the crux of the question - whether statistical procedures from different methods can be used interchangeably - I guess they can, as you highlight, but with good reason. Upon reflection, it might seem strange to use the sample-size weights when a particular sampling variance is assumed following a transformation. But it can still be done, as you suggest. $\endgroup$ Aug 13, 2021 at 12:17
  • $\begingroup$ It makes the task of the meta-analyst more complex when there is no framework or accepted standard for reliability meta-analyses, though some have tried to establish this (see Sánchez-Meca et al., 2021; Research Synthesis Methods; 12(4), 516-536). $\endgroup$ Aug 13, 2021 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.