When does Simpson's Paradox "end"?

Question

Disclaimer: This is not a duplicate of How to resolve Simpson's paradox.

As given in this blog, the following is the data of people on the titanic:

This is the same data when divided on basis of gender:

A complete reversal of the results seen in the first table!

The question:

Isn't it possible that I were to find another parameter, say Height of Passenger, based on which the new results indicated a complete reversal of those in table 2 above?

For example, male passengers with a height of 5.5 feet traveling in third class have a higher rate of survival than male crew members with a height of 5.5 feet.

Where does this end? And if I can, in theory, repeatedly find parameters whose inclusion change or reverse results, then is it ever possible to consider any result safe to use for future calculations?

Answer 1

Yes, you are right, we can create situations where the conditional association of one variable with another will change for each additional covariate you control for. For a simple simulation, I suggest you look Dagitty's Simpson's Machine based on Pearl's paper.

However, the question you should ask yourself is the following: why are you worried that the marginal association is different from the conditional association? That's perfectly normal.

So when you ask

when is it ever possible to consider any result safe to use for future calculations?

It seems you are not looking for associations only, but for stable, structural relationships. The short answer for your question is that data by itself, no matter how big, cannot help you---you need structural knowledge. Regarding more about Simpson's paradox, this answer might help.

Answer 2

Yes, I think there can always be some unexplored factor that --- had we evaluated that factor --- would have changed our interpretation of the results. That's just a reality of imperfect knowledge. And particularly problematic in observational studies like the one described where the observations are not balanced. (That is, where there are unequal numbers of each sex in each class).

But we should take some solace in the fact that we have some opportunities to assess our data to the best of our abilities.

For this example, the odds ratio for the first table is 1.007, suggesting the difference in survival rate between the two classes is so tiny that we likely would not have considered it interesting. That is, the survival rate for each class is essentially 24%.

The upshot here is that I think this example is less an example of a paradox where the trend reverses, than an example of seeing nothing interesting in the first table, but finding something interesting when more information is added in the second table.

It's only when we have the information in the second table that we get some sense of the factors affecting survival.

Because the underlying question is about what we can conclude about the effect of Class on survival rate, I'll use logistic regression to answer this question.

##### Table 2 #####

Data = read.table(header=T, text="
Class  Sex  Survive  NotSurvive
Third  M     75      387
Third  F     76       89
Crew   M    192      670
Crew   F     20        3
")

Trials = cbind(Data$Survive, Data$NotSurvive)

model = glm(Trials ~ Class + Sex + Class:Sex,
            data = Data,
            family = binomial(link="logit"))

library(car)

Anova(model)

   ### Analysis of Deviance Table (Type II tests)
   ### 
   ### Response: Trials
   ###           LR Chisq Df Pr(>Chisq)    
   ### Class       13.510  1  0.0002373 ***
   ### Sex         88.568  1  < 2.2e-16 ***
   ### Class:Sex    8.502  1  0.0035472 **

Note that the interaction of Class and Sex is significant, suggesting that this is the effect that we should be paying attention to.

In the results below, prob is the probability calculated in the table in the question.

library(emmeans)

emmeans(model, ~ Class:Sex, type="response")

   ### Class Sex      prob         SE  df asymp.LCL asymp.UCL
   ### Crew  F   0.8695652 0.07022340 Inf 0.6645495 0.9573281
   ### Third F   0.4606061 0.03880395 Inf 0.3860325 0.5369860
   ### Crew  M   0.2227378 0.01417187 Inf 0.1961989 0.2517422
   ### Third M   0.1623377 0.01715628 Inf 0.1314483 0.198824

We can also use estimated marginal means to estimate what the survival rate for each of classes would be had the sexes been balanced in each class. Below, we see that in fact, the survival in Crew is meaningfully and statistically higher.

This is a different conclusion than we would have come to from using the information in the first table only.

emmeans(model, ~ Class, type="response")

   ###  Class      prob         SE  df asymp.LCL asymp.UCL
   ###  Crew  0.5802181 0.07605615 Inf 0.4284069 0.7182285
   ###  Third 0.2891697 0.02063485 Inf 0.2504569 0.3312222

The addition of the information on sex has improved our understanding, but, still, there could always be some other important factor we have failed to measure that would have changed our interpretation.