I am currently planning a reliability meta-analysis or reliability generalization study (Vacha-Haase, 1998). I write this question assuming a random-effects model has been selected and Cronbach's a is the reliability estimate.
There are several considerations that need to be made, including:
Whether to transform the reliability coefficients (e.g., using Fisher z, Hakstian-Whalen, or Bonett transformations) or the use raw estimates.
Whether to use the inverse variance or sample size as weights.
Which heterogeneity estimator to use (e.g., Hunter-Schmidt, DerSimonian-Laird, REML, etc.).
Traditionally, meta-analysts have used a single method (e.g., the Hunter-Schmidt method involves raw estimates, sample-size weights, and the Hunter-Schmidt estimator of heterogeneity vs. the Hedges-Vevea method, which involves z transformed estimates for correlation coefficients [and Hakstian-Whallen/Bonett transformed estimates for Cronbach's alpha] and inverse variance weights).
This is certainly how some statistical packages operate too (e.g., with the
metafor package in
R, you cannot request z transformed estimates
("ZCOR") and use the sample-size weighted variance (
vtype = "AV"); the packages overrides this selection with the default settings for
Update: This is not completely accurate - the sampling variance is dependent on the treatment of the distribution, and in this case, it does not make any difference whether the large scale or sample-size weighted variance is used, see Wolfang's answer below.
metafor does not work 'within' methods and is as flexible as other packages.
There may be certain instances where meta-analysts wish to combine aspects from different methods because certain statistical procedures suit their dataset (e.g., normality of their coefficient distribution, the range of sample sizes, etc.), they wish to run sensitivity analyses switching certain procedures, or they wish to compare different procedures in simulation studies.
My question is: is it statistically sound to combine statistical procedures from different meta-analytic methods? For example, if one were to transform their reliability coefficients, would they also need to calculate the sampling variance using the assumed procedures (e.g., inverse variance in the case of z transform vs. sample-size weighted variance, for instance)?
Update: Again, the example given here is based on the assumption that the treatment of the coefficient distribution and sampling variance estimation method are two separate things, which is not quite the case. A more appropriate question is whether different statistical procedures can be used for each decision numbered above, e.g., transformed coefficients + sample-size weights (vs. inverse variance)
I am not asking if certain methods or statistical procedures are better or worse than others (a discussion of this can be found here). Instead, I wish to know it if is statistically sound to combine aspects from different methods?