6
$\begingroup$

I am reviewing a dissertation that does a meta-analysis of studies looking at a certain behaviour in rodents.

The author analysed ~20 studies and did a meta-analysis. They then split those studies into sub-categories, depending on the specific methodology that was used in each study (the same thing can be measured in several different ways) and performed one meta-analysis per group.

Finally, they split the original set in a different way (depending on another factor) and performed some further multiple meta-analyses.

I am really not that familiar with meta-analyses, so I am wondering whether this is good practice, or whether they should have included methodology as a factor in their overall analysis instead? If splitting data into multiple subgroups should there be a multiple comparison correction?

$\endgroup$
2
  • $\begingroup$ just wondering. is this group in the algebra sense? or in the regular sense? seems regular but just double checking. (i saw this in hot network, and i remember i had this one statistics instructor who said something like - well it was in tagalog/filipino, so i'm not really sure - 'you know what i like about statistics? it has actual applications. unlike, say, group theory. oh wait. i think group theory has applications cryptography. hmm. (back to lesson)') $\endgroup$ – BCLC May 18 at 7:12
  • 1
    $\begingroup$ @BCLC no, sorry, this is group in the "regular" sense... as "a subgroup of studies = a subset of the total studies". :) $\endgroup$ – nico May 18 at 7:41
4
$\begingroup$

For simplicity let us assume there are two groups although the issues are the same for multiple groups. The choices are (a) to run an analysis on each group separately (b) to run an analysis on the whole data-set with a two-level factor as a moderator.

The advantage of option (b) is that you have more data and hence your estimate of $\tau^2$ will be more precise. With 20 studies it was probably not very precise anyway (suggest they give confidence intervals for $\tau^2$ or $I^2$). A slight disadvantage of (b) is that it assumes that $\tau^2$ has the same value in each group but there are options in software to relax that assumption. There is detail on how to do this in R using the metafor package here.

If the split was based on theory then there is no real multiplicity issue here but if it is data-driven then caution is obviously needed.

$\endgroup$
1
$\begingroup$

There definitely should be a multiple comparison correction.

Meta-analyses are...generally not of the highest quality, with regards to statistical soundness.

$\endgroup$
1
$\begingroup$

It depends whether the data obtained using different methodologies can be statistically compared. Normally in a MA you first show the results overall and then you run a sensitivity analysis e.g. stratified analysis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.