How to handle Simpson's paradox

Question

Simpson's "paradox" is a well-known phenomenon that can be counter-intuitive for beginners: it is possible, say, for a medical trial to reveal that a certain treatment is beneficial to men as a group and to women as a group, but harmful to humans in the aggregate.

My first question is: what do practitioners do in such cases? Will doctors recommend the above treatment to their patients or not? Or is the Simpson's phenomenon ipso facto indicative of insufficient sample size/significance level, and hence renders the trial inconclusive?

Finally, has anyone studied quantitative versions of Simpson's phenomenon?

“Has anyone studied quantitative versions of Simpson’s phenomenon?” Can you help me understand your question? I didn’t realize there’s a non-quantitative version of it. — Arya McCarthy, Apr 4, 2021 at 12:32
Here's what I meant: the non-quantitative version is simply the phenomenon that a random variable X can have positive expectation unconditionally and negative expectation conditional on each of two complementary events. A quantitative version would say something about how large this gap can be as a function of some properties of the underlying joint distribution. — Aryeh, Apr 4, 2021 at 13:58
Simpson's paradox is a statistical phenomenon. Its interpretation depends entirely on the question being asked. If "is this treatment beneficial to humans (as a group)" is legit, then the answer "no" is also legit, regardless if it is beneficial to "men or women when considered separately", which strictly speaking is a separate question, no matter how subtly. As with many things, the real problem is with the question one asks, not the answer. — Tasos Papastylianou, Apr 4, 2021 at 21:50
If a treatment is beneficial to a woman and it is beneficial to a man it is beneficial to everyone. So yes the problem is with the question. Asking "is the treatment beneficial if I refused to know the sex of the patients even though there is a sex variable in the dataset and it was never missing" is an inappropriate question. — Frank Harrell, Apr 4, 2021 at 22:58
Aryeh, I don't think your comment is correct. $E[X] = E[X|A] \Pr[A] + E[X|\lnot A]\Pr[\lnot A]$. I think Simpson's is often cast as $E[X|A] < E[X|\lnot A]$, yet for all $y$, $E[X|A,Y=y] > E[X|\lnot A, Y=y]$. You may like Pearl's note "Understanding Simpson's Paradox" which claims to resolve the question via causality: ftp.cs.ucla.edu/pub/stat_ser/r414.pdf — usul, Apr 5, 2021 at 3:32

Nick Cox · Accepted Answer · 2021-04-05 09:02:17Z

13

For the case in which all patient descriptors are in the correct part of a causal diagram, a necessary but not sufficient condition for which is that the descriptors are assessed at "time zero" or before, Simpson's "paradox" is nothing more than a failure to ask a specific enough question. Stay away from marginal treatment effects and instead condition on all available information that is consistent with causal pathways. In the case of age and sex it is seldom inappropriate to condition on them. Treatment effects should be conditional and respect information flow. Focus on making the best treatment decision for the one patient being treated.

edited Apr 5, 2021 at 9:02

Nick Cox

53.1k8 gold badges121 silver badges175 bronze badges

answered Apr 4, 2021 at 13:13

Frank Harrell

83.9k6 gold badges166 silver badges356 bronze badges

$\begingroup$ (+1 but playing devil's advocate here:) Policy makers will contest that we need evidence that the treatment is beneficial to the whole population. If the marginal is "bad" but the conditional is "good", presenting this is a pain; especially if we have multiple factors at play (e.g. age:ethnicity:region) and a relatively straightforward grouping is unavailable. $\endgroup$
– usεr11852
Apr 5, 2021 at 1:09
3

$\begingroup$ I think you'd be hard pressed to find a policy maker who proposes the use of a treatment that is bad for every member of the population even if its good "on the average". Perhaps it's the notion of "population" and the mix of population members that is problematic and should be dropped. Patients are treated individually and respond individually. $\endgroup$
– Frank Harrell
Apr 5, 2021 at 11:09
2

$\begingroup$ A blanket decision based on a blanket estimate will be the wrong decision. $\endgroup$
– Frank Harrell
Apr 5, 2021 at 12:53
1

$\begingroup$ Your answer seems to contradict your comment below my question. In your answer, you write "In the case of age and sex it is seldom inappropriate to condition on them" -- so it seems the sex variable is relevant. But in the comment, your write "If a treatment is beneficial to a woman and it is beneficial to a man it is beneficial to everyone." How do the two jibe together? $\endgroup$
– Aryeh
Apr 5, 2021 at 13:41
2

$\begingroup$ I don't see any conflict between those two statements. You only learn about the truth about everyone if you condition on sex. Blanket estimate = unconditional estimate. $\endgroup$
– Frank Harrell
Apr 5, 2021 at 14:16

| Show 1 more comment

Christian Hennig · Accepted Answer · 2021-04-04 13:32:57Z

"What do practitioners do in such cases?" The key thing is to understand why, in the specific situation, Simpson's paradox arises. This depends on the situation. Let's imagine a medical trial example in which there are men, and women, treatment and placebo, "improvement" or "no improvement/harm". It may be that women are generally far more likely to show improvements, and also for some reason women received much placebo and little treatment. In this situation placebo may look inferior to treatment for both men and women, but better after aggregation. It is very important now why it happened that relatively more women were in the placebo group. If this was a randomised trial and it's just because of random variation of assignment, whereas the difference between men and women is meaningful, surely one should go by the men and women results individually (because the aggregation difference is not meaningful whereas the men/women difference is). However, one could also imagine a situation in which in fact for ethical reasons (probably not a valid thing to do in a clinical trial but anyway) there are a number of severe cases and it was decided that those are all given treatment, and almost all of these were men, and that among the non-severe cases there are no meaningful differences between men and women. Then one cannot say the treatment is better for both men and women, therefore better overall (despite the aggregate) - however one would need to take into account severity and couldn't make a conclusion by just looking at the aggregate either (because once more things are aggregated that are essentially different). One can also imagine a situation in which the differences between the groups involved in the paradox are not meaningful, and therefore the aggregate is more relevant, although that's more difficult (as there need to be systematic differences between the groups in order for the paradox to work - the situation needs to be constructed in such a way that these systematic differences are irrelevant to the study aim - unlikely in clinical trials, more likely maybe in social sciences where randomisation cannot be done and administrative decisions peripheral to the study aim may play a role).

In any case the baseline is that the occurrence of Simpson's paradox needs to be explained from background information, and what to do depends crucially on that explanation.

"Or is the Simpson's phenomenon ipso facto indicative of insufficient sample size/significance level, and hence renders the trial inconclusive?" No, in principle it can occur at any sample size, increasing it will not necessarily make it go away. One exception is if it occurs in a randomised trial due to an imbalance in groups caused by freak random numbers - this may be balanced out with a larger sample (once more it is important to understand what caused it).

Re "One can also imagine a situation in which the differences between the groups involved in the paradox are not meaningful, and therefore the aggregate is more relevant..." - Well, in real life treatment of people, randomization isn't done. Pharmacovigilance gives you tons of data on adverse effects of drugs, sometimes with a Simpson's paradox. Yet, I can't imagine how you could ever claim the factors to be "not meaningful" and why'd you'd ever treat your next patient based on the aggregate. — Jirka Hanika, Apr 5, 2021 at 21:57
As I wrote, unlikely in clinical trials. First time I was confronted with Simpson's paradox was about a university with two different campuses to both of which relatively more white than black people were admitted, but overall more black people. Now imagine that admission is actually done centrally and then students are assigned to campuses for reasons that are irrelevant regarding whether discrimination happens regarding admissions. I do agree that in most cases conditioning makes sense, but not always. — Christian Hennig, Apr 5, 2021 at 23:46
Standard therapy is not the same thing as clinical trials. And you again mention "imagine [unspecified] reasons that are irrelevant", while I am struggling to imagine any. That is, specific reasons, specific numbers, relevant sample sizes, full fledged Simpson's. The best example I can imagine is data that you can't use because you know that it's untrustworthy or far too small to be able to control for anything at all. — Jirka Hanika, Apr 6, 2021 at 7:21
How can you know that certain reasons are irrelevant when the data shows that they are relevant? — Jirka Hanika, Apr 6, 2021 at 7:23
It all depends on background, the full story, and all available information, so it's a bit pointless to discuss this based on made up examples. If you can't imagine situations of this kind in a certain area, so be it. "How can you know that certain reasons are irrelevant when the data shows that they are relevant?" In my example, it may be possible to find out the administrative reason why distributions of the two campuses are imbalanced, and they may be irrelevant to the issue of interest, which is discrimination (say). — Christian Hennig, Apr 6, 2021 at 10:25

Nat · Accepted Answer · 2021-04-07 03:24:21Z

tl;dr– Simpson's paradox isn't problematic unless correlations are inappropriately assumed to be causative.

Background: Simpson's paradox only happens when causation was fallaciously assumed.

Let's suppose these premises:

Healthcare can help apparently-healthy people.
Healthcare can help apparently-not-healthy people.
People receiving healthcare are more likely to be apparently-not-healthy.

Here we have an opportunity for Simpson's paradox: the bulk population has a negative association between apparent-health and receiving healthcare, despite each of the two sub-populations having a positive association.

But it's not really paradoxical (confusing), right? I mean, obviously, the negative association is due to people seeking medical attention when they're apparently-not-healthy.

The missing component is assuming causation. For example, if someone does a study on the correlation between apparent-health and receiving-healthcare, then assumes it causative, then they'd perceive a paradoxical situation in which:

Healthcare helps healthy people.
Healthcare helps not-healthy people.
Healthcare hurts (healthy OR not-healthy) people.

Absent false presumptions of causation, Simpson's paradox isn't a paradox.