10
$\begingroup$

The following is a question about the many visualizations offered as 'proof by picture' of the existence of Simpson's paradox, and possibly a question about terminology.

Simpson's Paradox is a fairly simple phenomenon to describe and to give numerical examples of (the reason why this can happen is deep and interesting). The paradox is that there exist 2x2x2 contingency tables (Agresti, Categorical Data Analysis) where the marginal association has a different direction from each conditional association.

That is, comparison of ratios in two subpopulations can both go in one direction but the comparison in the combined population goes in the other direction. In symbols:

There exist $a,b,c,d,e,f,g,h$ such that $$ \frac{a+b}{c+d} > \frac{e+f}{g+h} $$

but $$ \frac{a}{c} < \frac{e}{g} $$ and

$$ \frac{b}{d} < \frac{f}{h} $$

This is accurately represented in the following visualization (from Wikipedia ):

simpson' paradox in vectors

A fraction is simply the slope of the corresponding vectors, and it is easy to see in the example that the shorter B vectors have larger slope than the corresponding L vectors, but the combined B vector has smaller slope than the combined L vector.

There is a very common visualization in many forms, one in particular at the front of that wikipedia reference on Simpson's:

contrary slopes in sub populations

This is a great example of confounding, how a hidden variable (which separates two sub populations) can show a different pattern.

However, mathematically, such an image in no way corresponds to a display of the contingency tables that are at the basis of the phenomenon known as Simpson's paradox. First, the regression lines are over real-valued point set data, not count data from a contingency table.

Also, one can create data sets with arbitrary relation of slopes in the regression lines, but in contingency tables, there is a restriction in how different the slopes can be. That is, the regression line of a population can be orthogonal to all the regressions of the given subpopulations. But in Simpson's Paradox the ratios of the subpopulations, though not a regression slope, cannot stray too far from the amalgamated population, even if in the other direction (again, see the ratio comparison image from Wikipedia).

For me, that is enough to be taken aback every time I see the latter image as a visualization of Simpson's paradox. But since I see the (what I call wrong) examples everywhere, I'm curious to know:

  • Am I missing a subtle transformation from the original Simpson/Yule examples of contingency tables into real values that justify the regression line visualization?
  • Surely Simpson's is a particular instance of confounding error. Has the term 'Simpson's Paradox' now become equated with confounding error, so that whatever the math, any change in direction via a hidden variable can be called Simpson's Paradox?

Addendum: Here is an example of a generalization to a 2xmxn (or 2 by m by continuous) table: percentage baskets made with respect to distance, hidden variable is type of shot

If amalgamated over shot type, it looks like a player makes more shots when defenders are closer. Grouped by shot type (distance from basket really), the more intuitively expected situation occurs, that more shots are made the further away defenders are.

This image is what I consider to be a generalization of Simpson's to a more continuous situation (distance of defenders). But I still don't see yet how the regression line example is an example of Simpson's.

$\endgroup$
  • 5
    $\begingroup$ Simpson's Paradox does not apply only to categorical target data. Continuous target data with a categorical factor affecting it, as in your final graph, can be subject to the paradox. The key is that "categorical factor", not whether or not the variable of interest is categorical, or whether any or all of the other factors affecting the variable of interest are categorical. $\endgroup$ – jbowman Nov 29 '17 at 17:21
  • $\begingroup$ @jbowman OK, I can see possibly that SP might be generalizable beyond categorical data to continuous (I haven't seen that generalization; SP seems to be always presented with contingency tables), but I don't see how the second graph corresponds. I mean I see the obvious but vague metaphor "a hidden variable can change the direction", but I just don't see how the generalization works mathematically/precisely. $\endgroup$ – Mitch Nov 29 '17 at 22:19
  • 1
    $\begingroup$ You have a hidden categorical factor that causes the "real" data to follow the two colored lines, but without knowledge of it the data appears to follow the dotted line. Consider driving accidents by age as your target and x-axis variables - not categorical. They appear to go down with age, right? Now add the "hidden factor" of "driving while drunk". The blue line would be "driving while drunk", the red "driving while not drunk". Given that hidden factor, correlated with youth, accidents go up with age! (Not the most realistic example, I have to admit, but it's the idea that counts...) $\endgroup$ – jbowman Nov 29 '17 at 22:27
  • $\begingroup$ @jbowman That just sounds like an explanation of confounding error rather than of SP. Maybe you are saying that SP and confounding are the same. But that sounds in the direction of an answer; maybe you could formalize it a little more and make the connection with SP more explicit (account mathematically for how the regression lines are somehow like the ratio comparisons in the contingency table case). $\endgroup$ – Mitch Nov 29 '17 at 22:37
  • 1
    $\begingroup$ I agree the contingency version is different in a couple ways from the regression example in your question. (1) The confounder variable isn't a covariate $x$ describing an individual sample, it's some proportion $p$ which differs between treatment and control group. In the kidney stone example the proportion of large-stone patients is different between the two groups and that causes the paradox. (2) In the kidney example, the treatment does not correlate to a change in the confounding variable, it's a separate effect. $\endgroup$ – Paul Dec 11 '17 at 19:43
8
+50
$\begingroup$

The paradox is that there exist 2x2x2 contingency tables (Agresti, Categorical Data Analysis) where the marginal association has a different direction from each conditional association [...] Am I missing a subtle transformation from the original Simpson/Yule examples of contingency tables into real values that justify the regression line visualization?

The main issue is that you are equating one simple way to show the paradox as the paradox itself. The simple example of the contingency table is not the paradox per se. Simpson's paradox is about conflicting causal intuitions when comparing marginal and conditional associations, most often due to sign reversals (or extreme attenuations such as independence, as in the original example given by Simpson himself, in which there isn't a sign reversal). The paradox arises when you interpret both estimates causally, which could lead to different conclusions --- does the treatment help or hurt the patient? And which estimate should you use?

Whether the paradoxical pattern shows up on a contingency table or in a regression, it doesn't matter. All variables can be continuous and the paradox could still happen --- for instance, you could have a case where $\frac{\partial E(Y|X)}{\partial X} > 0$ yet $\frac{\partial E(Y|X, C = c)}{\partial X} < 0, \forall c$.

Surely Simpson's is a particular instance of confounding error.

This is incorrect! Simpson's paradox is not a particular instance of confounding error -- if it were just that, then there would be no paradox at all. After all, if you are sure some relationship is confounded you would not be surprised to see sign reversals or attenuations in contingency tables or regression coefficients --- maybe you would even expect that.

So while Simpson's paradox refers to a reversal (or extreme attenuation) of "effects" when comparing marginal and conditional associations, this might not be due to confounding and a priori you can't know whether the marginal or the conditional table is the "correct" one to consult to answer your causal query. In order to do that, you need to know more about the causal structure of the problem.

Consider these examples given in Pearl: enter image description here

Imagine that you are interested in the total causal effect of $X$ on $Y$. The reversal of associations could happen in all of these graphs. In (a) and (d) we have confounding, and you would adjust for $Z$. In (b) there's no confounding, $Z$ is a mediator, and you should not adjust for $Z$. In (c) $Z$ is a collider and there's no confounding, so you should not adjust for $Z$ either. That is, in two of these examples (b and c) you could observe Simpson's paradox, yet, there's no confounding whatsoever and the correct answer for your causal query would be given by the unadjusted estimate.

Pearl's explanation of why this was deemed a "paradox" and why it still puzzles people is very plausible. Take the simple case depicted in (a) for instance: causal effects can't simply reverse like that. Hence, if we are mistakenly assuming both estimates are causal (the marginal and the conditional), we would be surprised to see such a thing happening --- and humans seem to be wired to see causation in most associations.

So back to your main (title) question:

Does Simpson's Paradox cover all instances of reversal from a hidden variable?

In a sense, this is the current definition of Simpson's paradox. But obviously the conditioning variable is not hidden, it has to be observed otherwise you would not see the paradox happening. Most of the puzzling part of the paradox stems from causal considerations and this "hidden" variable is not necessarily a confounder.

Contigency tables and regression

As discussed in the comments, the algebraic identity of running a regression with binary data and computing the differences of proportions from the contingency tables might help understanding why the paradox showing up in regressions is of similar nature. Imagine your outcome is $y$, your treatment $x$ and your groups $z$, all variables binary.

Then the overall difference in proportion is simply the regression coefficient of $y$ on $x$. Using your notation:

$$ \frac{a+b}{c+d} - \frac{e+f}{g+h} = \frac{cov(y,x)}{var(x)} $$

And the same thing holds for each subgroup of $z$ if you run separate regressions, one for $z=1$:

$$ \frac{a}{c} - \frac{e}{g} = \frac{cov(y,x|z =1)}{var(x|z=1)} $$

And another for $z =0$:

$$ \frac{b}{d} - \frac{f}{h} = \frac{cov(y,x|z=0)}{var(x|z=0)} $$

Hence in terms of regression, the paradox corresponds to estimating the first coefficient $\left(\frac{cov(y,x)}{var(x)}\right)$ in one direction and the two coefficients of the subgroups $\left(\frac{cov(y,x|z)}{var(x|z)}\right)$ in a different direction than the coefficient for the whole population $\left(\frac{cov(y,x)}{var(x)}\right)$.

$\endgroup$
  • 1
    $\begingroup$ It sounds like, in your view, Simpson's paradox refers to not only the possibility of a difference in marginal and conditional associations, but also the confusion about which one is "right" to use when interpreting the data? And Pearl shows that the causal structure is what we should use to decide this? $\endgroup$ – Paul Dec 12 '17 at 18:40
  • 2
    $\begingroup$ "Simpson's paradox is about conflicting intuitions when comparing marginal and conditional associations." I disagree here, Simpson's paradox specifically refers to a flip-of-sign when comparing crude to stratified results. $\endgroup$ – AdamO Dec 12 '17 at 18:41
  • 2
    $\begingroup$ @AdamO while most people use the extreme case of sign reversal as the "strict" definition of Simpson's paradox, Simpson's original example actually had no sign reversal. $\endgroup$ – Carlos Cinelli Dec 12 '17 at 18:46
  • 1
    $\begingroup$ @Paul that's exactly right. $\endgroup$ – Carlos Cinelli Dec 12 '17 at 18:46
  • 2
    $\begingroup$ @AdamO I think Pearl's explanation of why this was deemed a "paradox" and why it still puzzles people is plausible. In the simple case of (a) for instance, causal effects can't simply reverse like that. Hence, if we are thinking causally for both cases, we would be surprised to see such a thing happening --- and humans seem to be wired to see causation in most associations. $\endgroup$ – Carlos Cinelli Dec 12 '17 at 19:20
2
$\begingroup$

Am I missing a subtle transformation from the original Simpson/Yule examples of contingency tables into real values that justify the regression line visualization?

Yes. A similar representation of categorical analyses is possible by visualizing the log-odds of response on the Y-axis. Simpson's paradox appears much the same way with a "crude" line running against the stratum-specific trends weighted in distance according to the stratum referent log-odds of the outcome.

Here's an example with the Berkeley admissions data

enter image description here

Here gender is a male/female code, on the X-axis is the crude admissions log-odds for male versus female, the heavy dashed black line shows gender preference: the positive slope suggests a bias toward male admissions. The colors represent admission to specific departments. In all but two cases, the slope of the department-specific gender-preference line is negative. If these results are averaged together in a logistic model not accounting for interaction, the overall effect is a reversal favoring female admissions. They applied to harder departments more frequently than did males.

Surely Simpson's is a particular instance of confounding error. Has the term 'Simpson's Paradox' now become equated with confounding error, so that whatever the math, any change in direction via a hidden variable can be called Simpson's Paradox?

Briefly, no. Simpson's paradox is merely the "what" whereas confounding is the "why". The dominant discussion has focused on where they agree. Confounding may have a minimal or negligible effect on estimates, and alternately Simpson's paradox, while dramatic, may be caused by non-confounders. As a note, the terms "hidden" or "lurking" variable are imprecise. From an epidemiologist perspective, careful control and design of study should enable measurement or control of possible contributors to confounding bias. They need not be "hidden" to be a problem.

There are times in which point estimates may vary drastically, to the point of reversal, that does not result from confounding. Colliders and mediators are also change effects, possibly reversing them. Causal reasoning warns that for studying effects, the main effect should be studied in isolation rather than adjust for these as the stratified estimate is wrong. (It is akin to inferring, incorrectly, that seeing the doctor makes you sick, or that guns kill people hence people don't kill people).

$\endgroup$
  • $\begingroup$ So you would say that Simpson's original example is not a case of "Simpson's paradox"? $\endgroup$ – Carlos Cinelli Dec 12 '17 at 19:11
  • $\begingroup$ @CarlosCinelli what example would you be referring to? I do not have access to Simpson's 1951 paper, but given it is published in JRSS and has no reference to an applied example in the abstract, it seems a purely theoretical work. $\endgroup$ – AdamO Dec 12 '17 at 20:23
  • $\begingroup$ It's the numerical example on paragraphs 9 and 10, where he gives the same contingency tables with two different stories that would lead to two different causal interpretations. In that example there's no sign reversal, just marginal independence. $\endgroup$ – Carlos Cinelli Dec 12 '17 at 20:36
  • 2
    $\begingroup$ To see why the sign reversal is inconsequential here, just imagine a situation where a treatment shows an extremely strong association for both men and women, but shows only a tiny association in the population overall. This would still be paradoxical too most people, if interpreted causally. $\endgroup$ – Carlos Cinelli Dec 12 '17 at 20:39
  • $\begingroup$ @CarlosCinelli I would have said that was an example of confounding but not Simpson's paradox per se but I won't belabor the point, I think you've made a good argument and perhaps I was holding some incorrect assumptions about what was and was not the elusive phenomenon of Simpson's Paradox. $\endgroup$ – AdamO Dec 13 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.