4
$\begingroup$

I have a data-set with 32 effect size estimates- only 11 of which report a value for the continuous moderator of interest (the samples anxiety level). A complete case analysis (restricted to the 11 cases) shows that anxiety is a significant predictor of the effect sizes in meta-regression.

I would like to use imputation techniques to "fill in" the missing values to see if the relationship between anxiety and effect size (d) is still significant. If I do this using the "mice" function in R, it automatically selects d (the effect size) as a predictor to impute the plausible values of anxiety, as shown in the predictor matrix.

My issue is that this seems circular- I already know that anxiety predicts d, so using d to predict the missing values of anxiety seems to be "playing tennis without the net" and will surely artificially strengthen the relationship.

On the other hand- I don't know for sure that increased anxiety predicts d or whether increased anxiety is an outcome of the effect size- they may have a mutual influence. (d reflects a bias for threat, which literature suggests could be a cause or consequence of anxiety.....). This would seem to make using d in imputation more legitimate. Also, the recommendation seems to be to use all variables that will appear in the model applied after imputation (which will of course include d as the outcome) in the imputation process (van Buuren, 1999).

So given this issue, should I simply remove the effect size as a predictor of anxiety for imputation and instead rely on other demographic variables, or just random sampling of the observed data? Doing this also seems wrong, since this seems to generate a false "uncertainty" in the imputed data, given we have an idea of the relation between d and anxiety.

Any help or references to resolve this problem would be much appreciated. Please let me know if anything is unclear.

$\endgroup$
2
$\begingroup$

Multiple Imputation was designed to be able to separate the missing data problem and the analysis of data. Because of this, I think I would agree with van Buuren's suggestion to use all variables that will be used in the model. Using this, you may first solve the missingness problem, and then go on to analyze the data. Unfortunately, I can not shed more light on this. For more information, you may look into Flexible Imputation of Missing Data, van Buuren (2012).

$\endgroup$
  • $\begingroup$ Why not also use for imputation some variables beyond those planned for the model? $\endgroup$ – rolando2 Sep 11 '14 at 18:46
  • $\begingroup$ Good point! If I recall correctly, the general advice is just to include as much information as possible in the imputation process. That may have a limit, of course. For example, I have no idea what happens to the imputation process if we were to add (say) 1000 generated random normal variables, unrelated to everything else. $\endgroup$ – Kees Mulder Sep 12 '14 at 8:31
  • $\begingroup$ I agree about adding information - random variables don't qualify. $\endgroup$ – rolando2 Sep 12 '14 at 11:10
1
$\begingroup$

Your question is very interesting. My recommendation would be to go both ways and see whether you have substantial discrepancies.

Another reasonable approach would be to use multivariate meta-analysis after multiple imputation, in order to see whether the imputation process (including one outcome) impacts on the association with the other outcomes.

In any case, the between imputed sets variance component will inform you on how much the missing data influence the final meta-analysis results.

$\endgroup$
1
$\begingroup$

The impact of missing data (including for covariates) is a growing concern for meta-analytic methodologists (see Cooper & Hedges, 2009; Pigott, 2012).

In terms of how to proceed, analytically, Mike Cheung's metaSEM() package (see Cheung, 2015a) for R is one of the only options that I know of that has features which explicitly deal with the missing covariate data problem. His documentation here details how his meta3x()function uses full-information maximum likelihood (FIML) with a structural equation modelling "trick" to address covariate missingness. It looks slick; the only complication (vs. the package's other functions) is that users must specify at what level their moderator variables (and auxiliary variables, if any) are located. His book (Cheung, 2015b) also provides a nice overview of the missing data issue in meta-analysis, including a more thorough description of the FIML process his package uses.

I'm not aware of anyone commandeering multiple imputation (yet) to handle missingness in meta-analysis, as you are attempting here. While it seems reasonable, in principle, to expect MI to serve this purpose, I wonder/worry if/how your MI procedure is weighting each study while drawing information from the covariance of auxiliaries/effect sizes and your non-missing moderator values, to fill in the missing values.

Regarding your concerns about "playing tennis without the net", I think there are a few reasons why this shouldn't trouble you too deeply. For one, MI/FIML will not just take the association between your observed moderator variables and effect sizes/auxiliary variables into account when "filling in" missing values; it will also take the uncertainty/imprecision of that association into account, and the less complete data you have, the more modest/uncertain those "guesses" for MI will be.

Secondly, there's pretty clear evidence at this point that under conditions of Missing Completely At Random (MCAR) and Missing At Random (MAR), that MI and FIML trounce deletion methods and simple substitution methods, in terms of bias and statistical power (see Enders, 2010, for a review). True, more bias creeps in and recovery (and therefore statistical power) is worse when missingness is not random (MNAR), but this is true for all missing data treatments, and the amount of bias introduced is still less for FIML compared to these other methods.

Finally, consider an example of missing data "circularity": imagine you have very good reason to trust that there is a detectable and appreciable difference in height between men and women. Then you encounter a data set in which there is some missing height data, but you know the birth-sex of the individuals in question. Is it playing tennis without the net to take into account the known association between sex and height? Or is maximizing your use of previous knowledge to make more accurate and efficient inferences? Surely there is a line at some point and an accompanying grey-zone where the difference between the two uses becomes hazy. But I don't think your circularity concern is as clear cut as it first seems, because there will be cases where, to not use advanced missing data treatment techniques, would essentially require you to hurt the power/accuracy of your investigation for fear of benefiting from previous knowledge.

References

Cheung, M. W. L. (2015a). metaSEM: An R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5, 1521.

Cheung, M. W. L. (2015b). Meta-analysis: A structural equation modeling approach. Chichester: Wiley.

Cooper, H., & Hedges, L. V. (2009). Potentials and limitations. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 561-572). New York, NY: Russell Sage.

Enders, C. K. (2010). Applied missing data analysis. New York, NY: Guilford Press.

Pigott, T.D. (2012) Advances in Meta-Analysis. New York, NY: Springer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.