I am trying to conduct a retrospective study to understand the effect of the presence of heart disease (N and Y) on binary categorical outcomes of certain types of surgeries (let's call them A, B, C). I'd also like to be able to compare the magnitude of these effects across surgeries.

If I do an independent analysis within each surgery group (e.g. perform Propensity Score Matching on heart disease within patients receiving surgery A, find odds ratio of heart disease), my gut feeling is that the odds ratios are not comparable across these groups since there was no matching performed across groups.

Main Q: How should I go about yielding effect magnitudes that are comparable?

I've tried treating the above scenarios as effectively six different treatments (NA, YA, NB, YB, NC, YC).

I've looked into matching across multiple treatment groups from: A Tutorial on Propensity Score Estimation for Multiple Treatments Using Generalized Boosted Models (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3710547/). However, the primary metric estimated is the Average Treatment Effect (ATE) which is defined as $$ E(Y [t′]) − E(Y [t″]) = \mu_{t'} - \mu_{t''}$$ where $Y[t'']$ is the outcome variable of interest under a different treatment $t''$ for patients who were originally treated under $t`$. So in essence, I'd like to compare $\mu_{NA} - \mu_{YA}$ vs $\mu_{NB} - \mu_{YB}$ vs $\mu_{NC} - \mu_{YC}$

My understanding is that this would mainly be applicable for continuous or ordinal outcome variables, as then it would make sense to compare outcome means of these two treatments. For binary categorical outcomes the interpretation would be comparing frequencies of the "Yes" categorical outcome.

This would be in contrast to comparing something like odds ratios between each group. Secondary Q: Should ATE be used for binary categorical variables, or should a different estimand/method be used?

Thank you for any help!


1 Answer 1


This is an issue of effect modification, or treatment heterogeneity across your subgroups of interest. In my opinion, a paper that fully describes the nuances in interpretation and estimation has not been written yet, but there are some papers that do a good job of describing the relevant issues and how to proceed. I particularly recommend Green and Stuart (2014).

You are correct in having some doubt about comparing ATEs between subgroups when the composition of those subgroups differs and thus they refer to different populations. For example, even if you remove confounding due to all background covariates (e.g., by matching or weighting or even in a randomized study), the distributions of patient characteristics across the surgery groups will differ. The challenge with this is that one cannot immediately disentangle the effect modification by the subgroup from the effect modification by patient characteristics that covary with subgroup.

An alternative approach is to consider surgery type its own kind of treatment, and treat the scenario as one of interaction. If it would make sense for any patient to receive any kind of surgery, then this type of analysis would be more appropriate than a simple subgroup analysis. In this way, you consider that you actually have 6 treatment groups (2 heart disease x 3 surgery type). You can use standard techniques for effect estimation with multi-category treatments and perform whatever contrasts are relevant for your analysis. This requires eliminating confounding to a larger degree, though, since you need to adjust not only for common causes of heart disease and the outcome, but also common causes of surgery type and the outcome. if you know the factors that lead to patients receiving different surgery types, this may not be much of an additional burden.

See VanderWeele (2009) for a nice discussion of the difference between effect modification and interaction.

Regarding whether it makes sense to report ATEs (i.e., risk differences) for binary outcomes, this depends on your purposes. If you want to estimate a marginal effect (i.e., the average effect of heart disease across the entire population), the risk difference is a nice, interpretable effect measure; risk ratios can be good too. Marginal odds ratios do not have a useful interpretation, so I would only recommend them if you had a specific reason to. If you want to estimate something more like the effect of heart disease for an individual, odds ratios have the nice property of there being the possibility that the conditional effects are equal across subgroups of other patient characteristics, whereas this is impossible with risk differences and risk ratios (i.e., because the effect will be smaller for people with a low baseline risk since there is only so low the risk can go). Marginal effects do not really help inform practice except when performed in narrow subgroups, whereas conditional effects are often estimated with bias and require unlikely assumptions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.