Alternative Aggregation Method to Avoid Simpson's Paradox

Question

The table below shows an example of Simpson's Paradox arising from some fictitious data for the success of operations performed by two doctors from the Simpsons TV show.

	Dr Hilbert	Dr Nick
Heart Surgery	70/90 (77%)	2/10 (20%)
Band Aid Removal	10/10 (100%)	81/90 (90%)
Total	80/100 (80%)	83/100 (83%)

Thinking about the data in terms of 4 regions:

A | B
-----
C | D

, I'm assuming, although I'm not quite sure how to prove it, that

(A > B) & (C > D) iff (A + C) > (B + D).

It seems the paradox arises due to the aggregation step - A + C are added as if when adding fractions, a/b + c/d were equal to (a + b) / (c + d).

What would be the correct way to aggregate the columns to get a better sense of the composition of the data? For example:

	Dr Hilbert	Dr Nick
Heart Surgery	70/90 (77%)	2/10 (20%)
Band Aid Removal	10/10 (100%)	81/90 (90%)
Total	160/90 (177.78%)	99/90 (110%)

Even though mathematically impossible, does this provide a better insight into the competence of Dr Hilbert than the original table (ignoring issues like selection bias due to expertise for now)?

Is this approach of taking the sum of the percentages a useful way to get a clearer picture of the aggregate results in cases where Simpson's Paradox is present, or can it be as misleading as the results provided by the usual method of aggregation?

Answer 1

1. It does not follow from (A+C) > (B+D) that A>B and C>D, as your "iff" line suggests. If A is much bigger than B, C can be smaller than D.

2. The issue here is that there is no uniquely "correct" way of aggregation. The "paradox" arises from the fact that there are different ways to look at the data that in some cases seem to contradict each other, however examples can be constructed in which they all make sense. In this example it might be that Dr Nick is actually the better doctor, therefore gets all the difficult cases assigned, and therefore has worse rates, but if Band Aid Removal were much more difficult (just as an example), achieving 81/90 involving all the difficult cases wouldn't be bad at all and the more telling thing would be that he got 90 of these cases assigned in the first place. Admittedly in the given example it is somewhat far fetched to argue that Dr Nick is better based on the numbers shown, however there are certainly cases in which the "overall sum" is important for comparison, whereas the distinction of the performance of the two performers into two categories has administrative reasons that are rather irrelevant for performance measurement, yet could explain a Simpson's paradox situation.

3. Despite this, I'd think that a situation in which performance measurement should be differentiated between the two categories (and then saying that Dr Hilbert is in fact better than Dr Nick because he's better in both categories despite not being better according to the overall sum) is the more frequent one. But nobody stops you from using aggregates for performance measurement other than the overall sum. Surely you can compute average ratios (88.5% for Hilbert, 55% for Nick) or weighted averages (in case that one of the two is more difficult). There's nothing "mathematically wrong" or impossible about this.

4. The baseline is that writing down the table and computing the overall sums is one thing, designing an appropriate measurement of performance is another. The sums are what the sums are, nobody can deny that or define "alternative sums". However, the "paradox" arises from the belief that these sums automatically carry with them some kind of interpretation (namely being some kind of "correct aggregate measure"). But this is not a mathematical fact, and in fact abstracting from the meaning and the background nothing can be said about what aggregation would be "correct". The sums as they are don't have any authority for decision making. Aggregates used as "performance measurement" have to be constructed taking the subject matter information into account, and there are no mathematical constraints as long as you can argue convincingly that this makes sense in the given situation.