I already made this question before (Is it appropriate to perform a meta-analysis of comparisons involving several communities where one of the communities was part of multiple studies?). I am making it again because after reading about network meta-analysis, as suggested in the answer that I received, I am not sure it would be helpful for my problem even if I had many studies at hand for the meta-analysis.

Here I try to elaborate my question more clearly.

I am conducting a systematic review about the effects of a policy aimed at the population level to prevent hospitalizations in older adults. The policy in question is the vaccination of children to prevent the spread of infections to older adults. In this case, the effect of the policy only affects the elderly indirectly because they are not themselves the target of the policy.

For one outcome I have 3 studies comparing communities where such policy was in place with other communities where it wasn’t.

Let’s name the communities where that policy was applied A and B, and the control communities as C, D and E.

The first study compared communities A versus C during year 2005.

The second study compared communities B versus D, during year 2005.

The third study compared communities A versus E, during years 2005, 2006 and 2007.

Please, note those are communities, not types of treatments. Policies enacted in communities A and B were similar, and communities C, D and E all represent the same absence of such policy.

I intended to perform a meta-analysis using the data from all three studies, using the estimates of all years available, but I am not sure if it would be appropriate because community A is present both in the first study and in the third study for the same year (2005).

I don’t think that the assumption of independence among studies needed for network meta-analysis holds for this case. I also understood network meta-analysis is useful to compare multiple treatments and to derive indirect comparisons between two treatments and not between two communities.

Because fixed-effect meta-analysis assume that there is a common-effect among studies and that all differences in observed effects are due to sampling error, I understand that it acknowledges to some extent the possibility of certain dependence between the effect estimates of different studies. Since fixed-effect meta-analysis is considered a special case of the random-effects meta-analysis I understand that such dependence would also be allowed in random-effects meta-analyses.


1 Answer 1


One of the points we like to teach is that you are not required to do a meta-analysis, but if you do, you need to justify undertaking one. In this case, you would have a very tough argument to demonstrate that a pooled effect estimate represents the 'truth'. Here are my thought on why that would prove to be difficult:

1) It seems that your data comes from observational data and you have not described any sort of adjustment for confounding factors. Therefore, any effect estimate could be attributed (at least in part) to effects unrelated to the intervention policy.

2) The allocation of participants was performed on a community level (cluster) rather than on an individual level. Therefore, the results have to be adjusted using the intra-class correlation coefficient. Most cluster studies I have seen don't seem to do that and act as if there is independence in the units analyzed (e.g. x number of people had an event in each group).

3) While study 1 and 2 can theoretically be pooled together. Study three has two main issues that make it troublesome to include in the analysis. The first is that one of the group is already included as a comparator in another study. This would lead to double-counting. This can be reduced by dividing the sample size of 'community A' in each comparison by half. Even if you did this, the range of years that are being reviewed are different in study 3 than the other studies. You would need to make some assumptions that I doubt would be valid to include it with the other studies in the same analysis.

Additionally, just to confirm, there is place for a network meta-analysis in this example as you are comparing two policies (not three). Also you have too few studies to conduct a meta-regression or any sort of advanced modeling. So what you're left with is an apple and an orange.

If you could get more data (maybe even possible de-identified raw data) from the study authors then you could analyze the data in a more practical fashion. If not, you'll have a hard time convincing skeptics that pooling was appropriate.

My five cents.

  • $\begingroup$ Many thanks for your careful answer! Yes, the studies I was trying to pool were observational ecological studies. I understand that results from cluster randomized trials should be analyzed taking the ICC into account to avoid the "unit of analysis error". But I am confused as to whether ecological observational studies also require such an approach because all inferences are made on the ecological / aggregate level. Because I was using the generic inverse variance method, I was only entering the log of RR and standard error in the meta-analysis, it did not feel like counting community A twice $\endgroup$
    – user156625
    Apr 24, 2017 at 17:04
  • $\begingroup$ I guess that my main doubt was exactly that. If by including both study 1 and 3 it would mean counting the community A twice for one year instead of just the effect estimate and the standard error of the comparisons between two sets of communities that involve A and other different communities. $\endgroup$
    – user156625
    Apr 24, 2017 at 17:32
  • $\begingroup$ While reading Diez-Roux article on multilevel analysis (biostat.jhsph.edu/~ejohnson/DiezRouz2000.pdf) I found the following statement "A second option is to focus exclusively on inter-group variation and on data aggregated to the group level. This approach eliminates the nonindependence problem mentioned above, but has the drawback of ignoring the role of individual-level variables in shaping the outcome". I am having a hard time finding references about meta-analysis of ecological studies... I understand that there are many possible biases involved in ecological studies, (continues $\endgroup$
    – user156625
    Apr 24, 2017 at 17:43
  • $\begingroup$ but I believe it should be possible to pool results from ecological studies considered similar enough. I agree that it is not mandatory to perform meta-analises just because data were extracted and that if one chooses to do so it must be explained why studies were considered similar enough. However I also believe that one should explain why meta-analyses were not performed and I am not sure that the fact that studies were ecological or observational is enough of a justification. The Cochrane Handbook even states that meta-analyses of observational studies should use adjusted data, recognizing $\endgroup$
    – user156625
    Apr 24, 2017 at 17:48
  • $\begingroup$ that different observational studies will have used different sets of covariates for adjustment (section Therefore, I am not completely convinced that I am comparing apples with oranges. That said (all comments above) @abousetta, I am really grateful that you took your precious time to answer my question and I would be really happy to listen to your thoughts about my comments above. $\endgroup$
    – user156625
    Apr 24, 2017 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.