Pooling homogenous studies vs. using meta-analysis/bayesian

Question

I am working with a student who has collected about 300 participants for his thesis. After collecting 100 participants, he analyzed the data. I had just recently read of a "mini-meta-analysis" strategy where someone collects pilot data, computes the estimates of interest, then performs a larger study and aggregates the estimates later using meta-analysis (Cumming, 2014). I suggested the student do something similar: run the analysis an additional time and use meta-analysis to pool the estimates.

The student asked what would be the advantage of using a meta-analysis relative to just analyzing all 300 participants. I couldn't provide a good answer (unless there was a lot of heterogeneity between the samples, which there isn't).

So, now to my question: if we have the raw data from 2+ homogenous studies, is there an advantage to computing the estimates twice then meta-analyzing them vs. computing them once?

My first impressions are:

1. The first study allows you to "calibrate" the model, then cross-validate in the second model. However, in this case, his statistical analysis was strictly confirmatory (and no "calibration" was required, aside from computing parameter estimates).

2. Having two samples allows us to do empirical bayesian analysis (we can use the parameters estimated in sample 1 as priors in sample 2 and take advantage of the strengths of bayesian statistics).

Is there another reason to favor one strategy over another?

---

Cumming, G. (2014). The new statistics: Why and how. Psychological science, 25(1), 7-29.

Answer 1

I will address frequentist meta-analysis, for which the answer is: The two approaches will give asymptotically equivalent point estimates if (1) you are using an effect size measure satisfying a property I will give below; and (2) the samples are homogenous not only in true effect size, but also in within-study variance.

Let $\mathbf{X}$ be your entire sample of size 300, and let $\mathbf{X}_1$ and $\mathbf{X}_2$ be the first and second parts of this sample (of sizes 100 and 200).

Let $\widehat{y}_i$ with $i \in {1,2}$ be point estimates from each sample $\mathbf{X}_1$ and $\mathbf{X}_2$. They have within-study variances $\sigma_i^2$, assumed fixed and known (the usual assumption in meta-analysis), and assumed equal by homogeneity. Let $\widehat{y}_R$ be the pooled point estimate from a random-effects meta-analysis, and let $\tau^2$ be the estimated heterogeneity (the variance of the true effects). Suppose you're meta-analyzing a statistic $g(\cdot)$.

Let's check whether the meta-analytically pooled point estimate matches the simple estimate pooling all the data (call it the aggregate-data estimate).

$$\eqalign{ \widehat{y}_R &:= \frac{ \sum_{i=1}^2 \frac{1}{\tau^2 +\sigma^2_i }\widehat{y}_i }{\sum_{i=1}^2\frac{1}{\tau^2 +\sigma^2_i }} \\ &\to \frac{ \sum_{i=1}^2 \frac{1}{\sigma^2_i }\widehat{y}_i }{\sum_{i=1}^2\frac{1}{\sigma^2_i }} \tag{$\tau^2 \to 0$, the truth}\\ &= \frac{ \frac{1}{\sigma^2}\widehat{y}_1 + \frac{1}{\sigma^2}\widehat{y}_2}{\frac{2}{\sigma^2}} \\ &= \frac{ \widehat{y}_1 + \widehat{y}_2}{2} }$$ (The penultimate line comes from assuming heterogeneity within-study variances.)

Now, this last expression is a simple average of the two samples' point estimates. So that means that if you choose a test statistic such that:

$$\frac{ g(\mathbf{X}_1) + g(\mathbf{X}_2) }{2} = g(\mathbf{X}) \tag{*}$$

then the meta-analytic estimate will be asymptotically equivalent to your aggregrate-data estimate. $(*)$ holds, for instance, if $g(\cdot)$ is the sample mean, but not if $g(\cdot)$ is the odds ratio. (However, note that you wouldn't want to meta-analyze untransformed odds ratios anyway since they do not fulfill the often-used normality assumption.)

Here is a code example to illustrate the equivalence when the statistic of interest is the sample mean. The inference also appears to be quite similar.

n1 = 100
n2 = 300 - n1
theta = 2  # true mean
sigw = 1  # common within-study variance

# generate whole dataset
X = rnorm( n1 + n2, mean = theta, sd = sigw )

# split into 2 subsamples
X1 = X[1:n1]
X2 = X[1:n2]

# get point estimates and SEs for each subsample
ests = c( mean(X1), mean(X2) )
ses = c( sd(X1) / sqrt(n1),  sd(X2) / sqrt(n2) )

# meta-analyze them
library(metafor)
ES = escalc( measure = "MD", yi = ests, sei = ses )
m = rma.uni(ES, method = "REML")

# compare point estimate to aggregate analysis
m$b; mean(X)

# compare inference to aggregate analysis
sqrt(m$vb); sd(X) / sqrt(n1 + n2)