How to handle Simpson's paradox Simpson's "paradox" is a well-known phenomenon that can be counter-intuitive for beginners: it is possible, say, for a medical trial to reveal that a certain treatment is beneficial to men as a group and to women as a group, but harmful to humans in the aggregate.
My first question is: what do practitioners do in such cases? Will doctors recommend the above treatment to their patients or not? Or is the Simpson's phenomenon ipso facto indicative of insufficient sample size/significance level, and hence renders the trial inconclusive?
Finally, has anyone studied quantitative versions of Simpson's phenomenon?
 A: "What do practitioners do in such cases?" The key thing is to understand why, in the specific situation, Simpson's paradox arises. This depends on the situation. Let's imagine a medical trial example in which there are men, and women, treatment and placebo, "improvement" or "no improvement/harm". It may be that women are generally far more likely to show improvements, and also for some reason women received much placebo and little treatment. In this situation placebo may look inferior to treatment for both men and women, but better after aggregation. It is very important now why it happened that relatively more women were in the placebo group. If this was a randomised trial and it's just because of random variation of assignment, whereas the difference between men and women is meaningful, surely one should go by the men and women results individually (because the aggregation difference is not meaningful whereas the men/women difference is). However, one could also imagine a situation in which in fact for ethical reasons (probably not a valid thing to do in a clinical trial but anyway) there are a number of severe cases and it was decided that those are all given treatment, and almost all of these were men, and that among the non-severe cases there are no meaningful differences between men and women. Then one cannot say the treatment is better for both men and women, therefore better overall (despite the aggregate) - however one would need to take into account severity and couldn't make a conclusion by just looking at the aggregate either (because once more things are aggregated that are essentially different). One can also imagine a situation in which the differences between the groups involved in the paradox are not meaningful, and therefore the aggregate is more relevant, although that's more difficult (as there need to be systematic differences between the groups in order for the paradox to work - the situation needs to be constructed in such a way that these systematic differences are irrelevant to the study aim - unlikely in clinical trials, more likely maybe in social sciences where randomisation cannot be done and administrative decisions peripheral to the study aim may play a role).
In any case the baseline is that the occurrence of Simpson's paradox needs to be explained from background information, and what to do depends crucially on that explanation.
"Or is the Simpson's phenomenon ipso facto indicative of insufficient sample size/significance level, and hence renders the trial inconclusive?" No, in principle it can occur at any sample size, increasing it will not necessarily make it go away. One exception is if it occurs in a randomised trial due to an imbalance in groups caused by freak random numbers - this may be balanced out with a larger sample (once more it is important to understand what caused it).
A: For the case in which all patient descriptors are in the correct part of a causal diagram, a necessary but not sufficient condition for which is that the descriptors are assessed at "time zero" or before, Simpson's "paradox" is nothing more than a failure to ask a specific enough question.  Stay away from marginal treatment effects and instead condition on all available information that is consistent with causal pathways.  In the case of age and sex it is seldom inappropriate to condition on them.  Treatment effects should be conditional and respect information flow.  Focus on making the best treatment decision for the one patient being treated.
A: tl;dr–  Simpson's paradox isn't problematic unless correlations are inappropriately assumed to be causative.

Background: Simpson's paradox only happens when causation was fallaciously assumed.
Let's suppose these premises:

*

*Healthcare can help apparently-healthy people.


*Healthcare can help apparently-not-healthy people.


*People receiving healthcare are more likely to be apparently-not-healthy.
Here we have an opportunity for Simpson's paradox: the bulk population has a negative association between apparent-health and receiving healthcare, despite each of the two sub-populations having a positive association.
But it's not really paradoxical (confusing), right?  I mean, obviously, the negative association is due to people seeking medical attention when they're apparently-not-healthy.
The missing component is assuming causation.  For example, if someone does a study on the correlation between apparent-health and receiving-healthcare, then assumes it causative, then they'd perceive a paradoxical situation in which:

*

*Healthcare helps healthy people.


*Healthcare helps not-healthy people.


*Healthcare hurts (healthy OR not-healthy) people.
Absent false presumptions of causation, Simpson's paradox isn't a paradox.

How to handle Simpson's paradox?
Same thing you do when you discover that $1 = 2 :$ realize that someone messed up along the way.
To be clear, a legitimate analysis can find correlations in which the total population has a different correlation than its component sub-populations, such as in the example above.  But a legitimate analysis shouldn't be able to arrive at a situation where causative behavior is reversed.
