I recently asked a question about Simpson's paradox. Suppose we are concerned about making some choice about an arbitrary element of a population. Recall that Simpson's paradox arises when the answer suggested by the aggregate data differs from the answer suggested when the data is split into two sub-populations and looked at separately. Refer to my previous question for details.
Both answers I got suggested I that side with the conclusion given by the sub-populations. They rightly remarked that the aggregate data is confounded and justified choosing the conclusion suggested by the sub-populations using a normalization scheme. An example normalization is given at the end of this answer.
The normalization approach as justification for siding with the sub-population conclusion is discussed in this Stanford Encyclopedia of Philosophy entry.
One way to arithmetically counter this difficulty is by ‘normalizing’ the representations of data from sub-populations and only pooling the normalized representations of the data. Normalizing data counters the effects of skewing by providing constant denominators for the fractions that represent the data, and by representing the sub-populations that are compared as if they were of equal sizes in the relevant respects in terms of which they are compared. However, Simpson's Reversals show that there are numerous ways of partitioning a population that are consistent with associations in the total population. A partition by gender might indicate that both males and females fared worse when provided with a new treatment, while a partition of the same population by age indicated that patients under fifty, and patients fifty and older both fared better given the new treatment. Normalizing data from different ways of partitioning the same population will provide incompatible conclusions about the associations that hold in the total population.
It seems, then, that siding with the conclusion suggested by the sub-population on this basis is completely arbitrary. Different partitions produce different conclusions. We can slice and the dice the data any way we like -- without prior information, there seems to be no principled way to prefer a conclusion suggested by a partition by age to the conclusion suggested by a partition by gender, for example.
Am I correct in thinking that using the normalization approach does not resolve Simpson's paradox in favor of the sub-population conclusion, for the reasons stated above?
Edit: I realize now that there's nothing particularly problematic about this. If you have access to both the age and gender splits, then use both to get the best result. I was concerned about a hypothetical case where Doctor A only had the age split and Doctor B only had the gender split, so they made different inferences. But it is not surprising this is the case, since they are inferring using different information.
Also it appears that I have slightly misinterpreted the answers I was given. My apologies.