Basic Simpson's paradox

Question

I have a question about something that my statistics teacher said about the following problem:

---

There are two hospitals named Mercy and Hope in your town. You must choose one of these in which to undergo an operation. You decide to base your decision on the success of their surgical teams. Fortunately, under the new health plan, the hospitals give data on the success of their operations, broken down into five broad categories of operations. Suppose you get the following data for the two hospitals:

Mercy Hospital
Type         A    B      C    D      E    All
Operations  359  1836   299   2086  149  4729
Successful  292  1449   179   434   13   2366
Hope Hospital 
Type          A   B  C   D   E   All
Operations   88 514 222 86  45   955
Successful   70 391 113 12  2    588

You notice that, in all types of operations, Mercy has a higher success rate than Hope, yet Hope has the highest overall success rate. Which hospital would you choose and why (choose two answers)?

A) Mercy; since I would go in for a specific operation, I want the hospital that has the best success rate for that operation.

B) Hope; since they do fewer operations in all categories, they are not "operation-happy" like Mercy.

C) Hope; this is an example of Simpson's paradox and we should always chose the "obvious" conclusion.

D) Mercy; looking at column E, Mercy clearly does more difficult surgeries and so is probably a better hospital.

E) Hope; it has the better overall success rate.

F) Mercy; this is an example of Simpson's paradox and we should always chose the opposite of the "obvious" conclusion.

---

My question isn't even about the occurrence of Simpson's paradox in this situation. My question is simply about the fact that my professor insists that A) and D) are the right answers instead of A) and F). He says,

"Because the success rate is so low for Type E surgeries,we can conclude that they are difficult and not just uncommon. Hence, Mercy probably has better equipment/doctors when compared to Hope."

I don't understand how he could imply on a statistical basis that he can tell that Mercy does "more difficult surgeries". It is obvious that Mercy has better success rate at type E surgeries, but why does that mean they do "more difficult surgeries". I think I am being screwed over by the wording of this problem and the professor isn't budging. Can someone please explain why I am wrong or how I can explain this to the professor?

Answer 1

I think A and E aren't a good combination, because A says you should pick Mercy and E says you should pick Hope.

A and D have the virtue of advocating the same choice. But, lets examine the line of reasoning in D in further detail, since that seems to be the confusion. The probability of success for the surgeries follows the same ordering at both hospitals, with the A type being most likely to be successful and the E type being the least likely. If we collapse over (i.e., ignore) the hospitals, we can see that the marginal probability of success for the surgeries is:

Type     A     B     C     D     E     All  
Prob   .81   .78   .56   .21   .08     .52

Because E is much less likely to be successful, it is reasonable to imagine that it is more difficult (although in the real world, other possibilities exist as well). We can extend that line of thinking to the other four types also. Now lets look at what proportion of each hospital's total surgeries are of each type:

Type     A     B     C     D     E  
Mercy  .08   .39   .06   .44   .03  
Hope   .09   .54   .23   .09   .05

What we notice here is that Hope tends to do more of the easier surgeries A-C (and especially B & C), and fewer of the harder surgeries like D. E is pretty uncommon in both hospitals, but, for what it's worth, Hope actually does a higher percentage. Nonetheless, the Simpson's Paradox effect is going to mostly be driven by B-D here (not actually column E as answer choice D suggested).

Simpson's Paradox occurs because the surgeries vary in difficulty (in general) and also because the N's differ. It is the differing base rates of the different types of surgeries that makes this counter-intuitive. What is happening would be easy to see if both hospitals did exactly the same number of each type of surgery. We can do that by simply calculating the success probabilities and multiplying by 100; this adjusts for the different frequencies:

Type     A     B     C     D     E     All  
Mercy   81    79    60    21    09     250  
Hope    80    76    51    14    04     225

Now, because both hospitals did 100 of each surgery (500 total), the answer is obvious: Mercy is the better hospital.

Answer 2

None of the answers are entirely baseless. But they ALL assume significant external knowledge and can't be taken to be correct strictly on the basis of the statistics.

A, B, D, and E all require assumptions about the factors the cause patients to choose one hospital over another; the process by which doctors and patients are matched up, the extent to which success rates are attributable to specific classes of operations vs. shared factors like ICU, and on an on.

In the real world we could legitimately consider many alternate factors such as the payment providers the hospital officially accepts, the socioeconomics and obesity rates of the neighborhood, whether this is a teaching hospital (in which case the success rate plummets when new interns arrive and we have to consider monthly mix), and on and on.

Obviously we can and do make reasonable assumptions about these factors, but without specifically addressing or excluding them from the problem, it's impossible to say if an answer is "right" or not.

Answer 3

@gung gave a very thorough answer, but there is one more reason why D is a correct answer to the question: Better hospitals do more of the difficult operations because they are better. That is, if a person comes into Hope Hospital for operation E (the hardest) they may send him/her to Mercy because they at Hope don't know how to do it.

This even happens in the real world, with the most difficult cases being sent to larger or more specialized hospitals.