Meta-analysis data from studies comparing the “ecological” effect of a policy when some communities are shared by different studies?

Question

I already made this question before (Is it appropriate to perform a meta-analysis of comparisons involving several communities where one of the communities was part of multiple studies?). I am making it again because after reading about network meta-analysis, as suggested in the answer that I received, I am not sure it would be helpful for my problem even if I had many studies at hand for the meta-analysis.

Here I try to elaborate my question more clearly.

I am conducting a systematic review about the effects of a policy aimed at the population level to prevent hospitalizations in older adults. The policy in question is the vaccination of children to prevent the spread of infections to older adults. In this case, the effect of the policy only affects the elderly indirectly because they are not themselves the target of the policy.

For one outcome I have 3 studies comparing communities where such policy was in place with other communities where it wasn’t.

Let’s name the communities where that policy was applied A and B, and the control communities as C, D and E.

The first study compared communities A versus C during year 2005.

The second study compared communities B versus D, during year 2005.

The third study compared communities A versus E, during years 2005, 2006 and 2007.

Please, note those are communities, not types of treatments. Policies enacted in communities A and B were similar, and communities C, D and E all represent the same absence of such policy.

I intended to perform a meta-analysis using the data from all three studies, using the estimates of all years available, but I am not sure if it would be appropriate because community A is present both in the first study and in the third study for the same year (2005).

I don’t think that the assumption of independence among studies needed for network meta-analysis holds for this case. I also understood network meta-analysis is useful to compare multiple treatments and to derive indirect comparisons between two treatments and not between two communities.

Because fixed-effect meta-analysis assume that there is a common-effect among studies and that all differences in observed effects are due to sampling error, I understand that it acknowledges to some extent the possibility of certain dependence between the effect estimates of different studies. Since fixed-effect meta-analysis is considered a special case of the random-effects meta-analysis I understand that such dependence would also be allowed in random-effects meta-analyses.

Answer 1

One of the points we like to teach is that you are not required to do a meta-analysis, but if you do, you need to justify undertaking one. In this case, you would have a very tough argument to demonstrate that a pooled effect estimate represents the 'truth'. Here are my thought on why that would prove to be difficult:

1) It seems that your data comes from observational data and you have not described any sort of adjustment for confounding factors. Therefore, any effect estimate could be attributed (at least in part) to effects unrelated to the intervention policy.

2) The allocation of participants was performed on a community level (cluster) rather than on an individual level. Therefore, the results have to be adjusted using the intra-class correlation coefficient. Most cluster studies I have seen don't seem to do that and act as if there is independence in the units analyzed (e.g. x number of people had an event in each group).

3) While study 1 and 2 can theoretically be pooled together. Study three has two main issues that make it troublesome to include in the analysis. The first is that one of the group is already included as a comparator in another study. This would lead to double-counting. This can be reduced by dividing the sample size of 'community A' in each comparison by half. Even if you did this, the range of years that are being reviewed are different in study 3 than the other studies. You would need to make some assumptions that I doubt would be valid to include it with the other studies in the same analysis.

Additionally, just to confirm, there is place for a network meta-analysis in this example as you are comparing two policies (not three). Also you have too few studies to conduct a meta-regression or any sort of advanced modeling. So what you're left with is an apple and an orange.

If you could get more data (maybe even possible de-identified raw data) from the study authors then you could analyze the data in a more practical fashion. If not, you'll have a hard time convincing skeptics that pooling was appropriate.

My five cents.