A question about resolving Simpson's paradox by normalization

Question

I recently asked a question about Simpson's paradox. Suppose we are concerned about making some choice about an arbitrary element of a population. Recall that Simpson's paradox arises when the answer suggested by the aggregate data differs from the answer suggested when the data is split into two sub-populations and looked at separately. Refer to my previous question for details.

Both answers I got suggested I that side with the conclusion given by the sub-populations. They rightly remarked that the aggregate data is confounded and justified choosing the conclusion suggested by the sub-populations using a normalization scheme. An example normalization is given at the end of this answer.

The normalization approach as justification for siding with the sub-population conclusion is discussed in this Stanford Encyclopedia of Philosophy entry.

One way to arithmetically counter this difficulty is by ‘normalizing’ the representations of data from sub-populations and only pooling the normalized representations of the data. Normalizing data counters the effects of skewing by providing constant denominators for the fractions that represent the data, and by representing the sub-populations that are compared as if they were of equal sizes in the relevant respects in terms of which they are compared. However, Simpson's Reversals show that there are numerous ways of partitioning a population that are consistent with associations in the total population. A partition by gender might indicate that both males and females fared worse when provided with a new treatment, while a partition of the same population by age indicated that patients under fifty, and patients fifty and older both fared better given the new treatment. Normalizing data from different ways of partitioning the same population will provide incompatible conclusions about the associations that hold in the total population.

It seems, then, that siding with the conclusion suggested by the sub-population on this basis is completely arbitrary. Different partitions produce different conclusions. We can slice and the dice the data any way we like -- without prior information, there seems to be no principled way to prefer a conclusion suggested by a partition by age to the conclusion suggested by a partition by gender, for example.

Am I correct in thinking that using the normalization approach does not resolve Simpson's paradox in favor of the sub-population conclusion, for the reasons stated above?

Edit: I realize now that there's nothing particularly problematic about this. If you have access to both the age and gender splits, then use both to get the best result. I was concerned about a hypothetical case where Doctor A only had the age split and Doctor B only had the gender split, so they made different inferences. But it is not surprising this is the case, since they are inferring using different information.

Also it appears that I have slightly misinterpreted the answers I was given. My apologies.

Answer 1

There is no one answer. The quote you use to counter the normalization proposal for your specific example seems to merely argue that the issue is complex. It says that a solution might be normalization but that there may be unanticipated ways of slicing the data (although I'm not a fan of the way this one is written). I don't see a statement there saying never to do it.

Simpson's paradoxes need to be taken on a case by case basis with careful analysis of the data, targeted questions about the data, and reasonable responses. Any treatise would therefore require some caution about normalization.

You argue that the prior answers you received rightly remarked that the sub population conclusion was best. That's how you make it no longer, as you say "seem arbitrary". You select the reasonable assessment of the data to tackle the question at hand. If more slices of the data you presented were available, or if different questions were being asked, then different answers might arise. None of them contradict that given the data as they were presented the sub population answer is the best response.

(BTW, it's your interpretation of responses to your prior question that they advocate normalization. Neither of them say to normalize. Both say look at the cells.)