# What to look for to spot potential cases of Simpson's paradox?

I understand the basics of Simpson's paradox, but I'm not yet confident that I'd always be able to "avoid" it, or to spot cases where others have failed to do so. To be more precise, I'm not sure I would always be able to spot situations in which some fallacious argument of the form

$$\forall i, P(A|C_i) > P(B|C_i) \Leftrightarrow P(A) > P(B),$$

or an equally fallacious variant thereof, is being invoked (most likely tacitly).

How often do such errors occur in the peer-reviewed literature? How often are authors called on them? Are there common but hard-to-detect forms of such arguments?

For example, I routinely come across "averages of averages" being treated as equivalent to "global averages". Is this a potential backdoor to Simpson's-paradox-like fallacies?

• Unless you have an oracle that can see when there's an important variable you haven't accounted for, I don't know that it's possible to have a foolproof way to know you avoided it - nor any reliable means of figuring out how often it happens. Well designed experiments, of course, are organized to avoid it (by randomization of treatment to subjects), but that's not always possible. The fallacious argument you wrote at the end there looks like it might also relate to some other aspects of the ecological fallacy as well. – Glen_b Nov 4 '14 at 4:41