Simpson's paradox is introduced on wikipedia using 'metrics' of success rates and regression coefficients, for the first (success rate of kidney stone treatments): How to resolve Simpson's paradox?, and the second: Coefficient changes sign when adding a variable in logistic regression, and also, this paper Understanding Simpson’s Paradox linked from the first question.
Quoting from the paper:
If we partition the data into subpopulations, each representing a specific value of the third variable, the phenomena appears as a sign reversal between the associations measured in the disaggregated subpopulations relative to the aggregated data, which describes the population as a whole.
It seems to me this quote still applies if we replace "sign reversal between the associations measured" such that:
If we partition the data into subpopulations, each representing a specific value of the third variable, the phenomena appears as a change in the interpretation of any metric relative to the aggregated data, which describes the population as a whole.
That is, the important bit about the paradox is actually the "partition the data into subpopulations" and not the specific metric. As with the example on kidney stones, the second quote makes sense if the metric is the success rate of a treatment.
In this question,
- I am wondering whether the second quote is an appropriate generalisation of Simpson's paradox beyond success rates and regression coefficients.
- Has anyone encountered other examples (not related to success rates or regression coefficients) you would consider as a type of Simpson's paradox?
For example, when evaluating two competing models using
log-likelihood mean model log-likelihood per observation, would it be possible that model A is better than model B on disjoint subsets $X$ and $Y$ of the data, but worse than model B on the whole dataset $X \cup Y$.