1
$\begingroup$

I am performing a meta-analysis and want to compute a summary effect (i.e. weighted mean) for studies that report data on more than one effect size. The same participants are involved, however the number of participants for each effect size may differ. To take into account the different sample sizes, the weight (W) assigned to each effect size (ES) would the inverse of that effect size’s variance (V), i.e.:

W = 1 / V

The weighted mean (M) of the combined effect sizes then would be:

M = ∑ ES x W / ∑ W

With variance of this new mean:

V = 1 / ∑ W

The summary effect (i.e. weighted mean) is used in the subsequent meta-analysis with the effect sizes (means) of other studies. To be able to do this, I need the sample size (N) for each effect size. My question therefore is: what would be the (new) sample size of a summary effect if the sample sizes for the effect sizes that together make up the summary effect are different, e.g. n=105, n=115, and n=120?

The method above assumes the combined effect sizes (within a study) are independent (r = 0). This is very unlikely. My second question, therefore, is: how can I incorporate the correlation between the effect sizes within a study and, at the same time, take into account the different sample sizes for these effect sizes (assigning weight to each effect size based on the inverse of that effect size’s variance)?

Thank you in advance for your help!

$\endgroup$

1 Answer 1

0
$\begingroup$

You cannot just treat $M$ as if it was an individual effect size with some new sample size. Here, you already have the sampling variance ($V$) for this (combined) estimate and that is what would go into the meta-analysis (the sample size would only be needed in order to compute the sampling variance, but you already have it).

In order to properly take the correlation between the effects into consideration, you would need to compute their covariances. See, for example, chapter 24 in Borenstein et al. (2009), Introduction to Meta-Analysis. Since the information needed to compute the covariances is often not available (specifically, the correlation between the variables underlying the different effect sizes that were computed), one could make a 'guestimate' about their correlation and then use that.

The equations for combining the effect sizes into a single aggregated effect size are as follows (using inverse-variance weighting): Let $V$ be the $k \times k$ var-cov matrix of the $k$ estimates in $y$. Then the combined effect size is $M = (X'WX)^{-1} X'Wy$, where $W = V^{-1}$ and $X$ is just a column vector of 1's. The sampling variance of $M$ is then $(X'WX)^{-1}$.

$\endgroup$
3
  • $\begingroup$ Thanks a lot for the clarifcation Wolfgang. I am actually using metafor and would like to use the combined estimates in a random effects model. To run the model the N of each study is however required. I am running the model for raw means, i.e.: model <- rma(measure="MN", mi=uni.mean, sdi=uni.sd, ni=N, data=Df). I assume the N is needed to run the model? $\endgroup$
    – Tidz
    Sep 21, 2021 at 18:08
  • $\begingroup$ You need to do this in two steps. First compute the outcomes (with escalc()), then feed those to rma(). Actually, you have a step in between where you combine multiple estimates from the same group. You can actually do this with aggregate(). See the docs for further details. $\endgroup$
    – Wolfgang
    Sep 21, 2021 at 22:09
  • $\begingroup$ I had not discovered the aggregate( ) function: wviechtb.github.io/metafor/reference/aggregate.escalc.html That appears to be function I am looking for. Possibly I will make the switch to a rma.mv model, but as the number of studies with multiple effect sizes for now is limited, the impact likely is limited. Thanks for your help. $\endgroup$
    – Tidz
    Sep 22, 2021 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.