Meta-analysis based on cohort studies only

Question

I am planning a meta-analysis based on cohort studies only. There are dozens of papers reporting results of treatments A and B. But the treatments were never compared with each other. Besides, there is no common reference treatment C to allow performing network meta-analysis of A vs B vs C; to be exact, there are too many reference treatments.

Thus, I have to use the results on treatments A and B as the results of single-arm studies, just ignoring other arms in the previous studies.

The outcome is binary. What are available methods for such a meta-analysis? I have found some answers in Network Meta-Analysis: Single Arm vs Contrasts Strategy Pros and Cons , but that question is somewhat different.

So, what am I to use? Mixed effects logistic meta-regression? Is it necessary to account for baseline characteristics in those single-arm studies if I am going to incorporate, say, 50 papers into meta-analysis?

Answer 1

A mixed effects logistic regression (if the outcome is binary/binomial) is certainly one plausible approach that could deal with the situation that all studies were essentially identical, in identical populations and there's just a bit of random variation in what you'd have expected to happen on a control group (if one had been in included in all studies). It is even a coherent method for combining randomized and single-arm studies.

However, there's some big issues with doing such an analysis that is not solely based on randomized comparisons. You'd definitely want to account for

• differences in study population (esp. those that might change the outcome you're looking at such as disease severity, background medications, age, baseline measurements related to disease state...) and
• time period when studies were conducted (are outcomes for newer treatment just better because recently patient outcomes are just better due to better medical care, shifts over time in how early diseases get diagnosed) and
• study design such as
  • inclusion criteria (if I enroll only those with a hypertension diagnosis and baseline blood pressure > 140 mmHg, I will get a larger within group change [for the same exact intervention] from baseline than if I recruit all with a hypertension diagnosis - the same if I look at achieving SBP <= 140)
  • treatment/study duration (if you turn the outcome into binary dead/not dead, then a longer study will just have a higher proportion of deaths)
  • allowed concomitant medications
  • outcome definitions/measurement methods...

Any one of these could otherwise due to the lack of randomized comparisons massively bias comparisons between single-arm outcomes (to the extent that they could make a completely ineffective intervention look better than a highly effective one). In a way, this might be a much worse scenario than an observational study, where you at least usually have a common data source with all data from the same time period and access to individual patient data (important confounders might still be unavailable).

Note that you don't need to account for just for the things you have data published for, but all the relevant things that might meaningfully matter. If you cannot get information on really important differences (or if some study design differences are completely confounded with evaluated intervention, e.g. because all studies for one intervention are old and all studies for a new intervention are new, and this leads to a complete confounding vs. substantially changed standard of care), you might just not be able to trust your analysis. If it's relatively few studies and a lot of potentially important differences between studies, one solution might be to set prior distribution in a Bayesian analysis (because you probably do have some a rough idea how much certain things could potentially change outcomes in what direction).

Answer 2

Do you have raw individual data with 0/1 responses in each study? If so, you can use each study as a group indicator in random intercepts and slopes in binary regression with mixed effects. If the studies have multiple measurements on each patient, you can also have two layers of nested random effects lme4::glmer(y ~ treatment * time + (1 | study/patient), family = binomial) or use conditional fixed-effect estimator such as survival::clogit in R. Baseline characteristics should be added as predictors even if treatment assignment was random because it will reduce the standard error of treatment effect. We should also add study design characteristics as predictors that can potentially affect the response and treatment effect, such as country, randomization, and screening. See an example meta-analysis Lehner, S., & Peer, S. (2019). The price elasticity of parking: A meta-analysis. Transportation Research Part A: Policy and Practice, 121, 177–191. https://doi.org/10.1016/j.tra.2019.01.014

If individual data are not available, it will be difficult to compare treatment efficacy. Binary regression coefficients from summary tables are not usable across different studies because they are standardized by an unknown factor that varies among different models. See Williams, R., & Jorgensen, A. (2023). Comparing logit & probit coefficients between nested models. Social Science Research, 109, 102802. https://doi.org/10.1016/j.ssresearch.2022.102802. We will need to translate model coefficients into average marginal effects on probability. This requires at least the variance-covariance matrix of model coefficients. Because logit and probit are nonlinear transformation, the effect of treatment on the event probability is also nonlinear in dosage and dependent on the value of other predictors. Thus, for binary regression models among different studies to be comparable, we need to state the effect in terms of the changes in event probability between treatment groups at specific values of other predictors (e.g., age, sex).

If only summary data are available, one feasible way is to model the collective response count as a binomial process where each study contributes two observations, one for Treatment A where event happens to n1 patients out of nA in the group and another for Treatment B where event happens to n2 patients out of nB in the group. If each patient was measured multiple times, more summary observations for each group should be added. This can still be modeled by a mixef-effects logit or probit model, such as lme4::glmer(cbind(case, total) ~ treatment + (1 | study), family = binomial).