I am working with a student who has collected about 300 participants for his thesis. After collecting 100 participants, he analyzed the data. I had just recently read of a "mini-meta-analysis" strategy where someone collects pilot data, computes the estimates of interest, then performs a larger study and aggregates the estimates later using meta-analysis (Cumming, 2014). I suggested the student do something similar: run the analysis an additional time and use meta-analysis to pool the estimates.

The student asked what would be the advantage of using a meta-analysis relative to just analyzing all 300 participants. I couldn't provide a good answer (unless there was a lot of heterogeneity between the samples, which there isn't).

So, now to my question: if we have the raw data from 2+ homogenous studies, is there an advantage to computing the estimates twice then meta-analyzing them vs. computing them once?

My first impressions are:

  1. The first study allows you to "calibrate" the model, then cross-validate in the second model. However, in this case, his statistical analysis was strictly confirmatory (and no "calibration" was required, aside from computing parameter estimates).

  2. Having two samples allows us to do empirical bayesian analysis (we can use the parameters estimated in sample 1 as priors in sample 2 and take advantage of the strengths of bayesian statistics).

Is there another reason to favor one strategy over another?

Cumming, G. (2014). The new statistics: Why and how. Psychological science, 25(1), 7-29.

  • $\begingroup$ The correct answer depends on the type of study you are conducting (eg intervention vs prediction). In addition, you should be wary of any scenario in which the all-in and the 2-step analyses are conflicting. My recommendation would be in any case, if the selection criteria and procedures are the same, to use a single analysis approach. Conversely, if there are even minor differences in selection or you aim for prediction, go first for a 2-step approach (but with individual patient-level meta-analysis). $\endgroup$ – Joe_74 May 23 '18 at 13:10
  • 1
    $\begingroup$ Thanks for the comment. This is a prediction, not an intervention study. The selection criteria and procedures are the same. I suppose we can analyze it both ways and (hopefully) the results show it doesn't matter. $\endgroup$ – dfife May 23 '18 at 18:28
  • $\begingroup$ Yes, do it both ways. But probably for prediction purposes the two-step strategy (with separate derivation and validation) is more meaningful. However, your sample might be not large enough for precise modeling. Check for instance this: springer.com/us/book/9780387772431 $\endgroup$ – Joe_74 May 24 '18 at 5:55

I will address frequentist meta-analysis, for which the answer is: The two approaches will give asymptotically equivalent point estimates if (1) you are using an effect size measure satisfying a property I will give below; and (2) the samples are homogenous not only in true effect size, but also in within-study variance.

Let $\mathbf{X}$ be your entire sample of size 300, and let $\mathbf{X}_1$ and $\mathbf{X}_2$ be the first and second parts of this sample (of sizes 100 and 200).

Let $\widehat{y}_i$ with $i \in {1,2}$ be point estimates from each sample $\mathbf{X}_1$ and $\mathbf{X}_2$. They have within-study variances $\sigma_i^2$, assumed fixed and known (the usual assumption in meta-analysis), and assumed equal by homogeneity. Let $\widehat{y}_R$ be the pooled point estimate from a random-effects meta-analysis, and let $\tau^2$ be the estimated heterogeneity (the variance of the true effects). Suppose you're meta-analyzing a statistic $g(\cdot)$.

Let's check whether the meta-analytically pooled point estimate matches the simple estimate pooling all the data (call it the aggregate-data estimate).

$$\eqalign{ \widehat{y}_R &:= \frac{ \sum_{i=1}^2 \frac{1}{\tau^2 +\sigma^2_i }\widehat{y}_i }{\sum_{i=1}^2\frac{1}{\tau^2 +\sigma^2_i }} \\ &\to \frac{ \sum_{i=1}^2 \frac{1}{\sigma^2_i }\widehat{y}_i }{\sum_{i=1}^2\frac{1}{\sigma^2_i }} \tag{$\tau^2 \to 0$, the truth}\\ &= \frac{ \frac{1}{\sigma^2}\widehat{y}_1 + \frac{1}{\sigma^2}\widehat{y}_2}{\frac{2}{\sigma^2}} \\ &= \frac{ \widehat{y}_1 + \widehat{y}_2}{2} }$$ (The penultimate line comes from assuming heterogeneity within-study variances.)

Now, this last expression is a simple average of the two samples' point estimates. So that means that if you choose a test statistic such that:

$$\frac{ g(\mathbf{X}_1) + g(\mathbf{X}_2) }{2} = g(\mathbf{X}) \tag{*}$$

then the meta-analytic estimate will be asymptotically equivalent to your aggregrate-data estimate. $(*)$ holds, for instance, if $g(\cdot)$ is the sample mean, but not if $g(\cdot)$ is the odds ratio. (However, note that you wouldn't want to meta-analyze untransformed odds ratios anyway since they do not fulfill the often-used normality assumption.)

Here is a code example to illustrate the equivalence when the statistic of interest is the sample mean. The inference also appears to be quite similar.

n1 = 100
n2 = 300 - n1
theta = 2  # true mean
sigw = 1  # common within-study variance

# generate whole dataset
X = rnorm( n1 + n2, mean = theta, sd = sigw )

# split into 2 subsamples
X1 = X[1:n1]
X2 = X[1:n2]

# get point estimates and SEs for each subsample
ests = c( mean(X1), mean(X2) )
ses = c( sd(X1) / sqrt(n1),  sd(X2) / sqrt(n2) )

# meta-analyze them
ES = escalc( measure = "MD", yi = ests, sei = ses )
m = rma.uni(ES, method = "REML")

# compare point estimate to aggregate analysis
m$b; mean(X)

# compare inference to aggregate analysis
sqrt(m$vb); sd(X) / sqrt(n1 + n2)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.