I'm not entirely sure what I am talking about is Simpson's paradox, because an opposite relationship does not appear when you combine two data sets, but merely a different one. Still, I think it is some variation of the paradox.

I present below some data I am working with, made generic as X and Y.

Here are the relationships in question uncombined:

enter image description here

When the two data sets are combined, a strong negative relationship appears between. Clearly, uncombined, there is not a strong negative relationship in both data sets. For my study, it's important that there is not the same strong negative relationship combined as there is uncombined.

What I want to know is there any kosher way to say—or to show statistically—in a publishable paper that a strong negative relationship may only appear when these datasets are combined due to the following: The datapoints for Y in the green set are generally higher than the Y datapoints in the yellow set, and the datapoints for X are generally smaller for green than they are for yellow?

I have asked my statistician working with me on this, and he does not know of any such way, but he advised me to seek out a way to do it, if feasible.

Here are the data for these two sets, color coded:

Y       X            Color
29.2    3.822954823  orange
45.4    4.446472019  orange
37.8    4.364963504  orange
18.6    4.154740061  orange
36.2    3.449355433  orange
22.2    4.426129426  orange
49.8    3.765931373  orange
28.6    4.552311436  orange
54.4    4.270718232  orange
49.4    4.668501529  orange
18      4.32480195   orange
41.6    3.733698964  orange
59.6    3.371865443  green
52.3    3.404674047  green
76.8    3.20353443   green
41.8    3.198529412  green
64.2    3.352293578  green
34.4    3.559021407  green
69.3    4.107033639  green
62.4    3.363302752  green
45      3.109489051  green
59.2    3.8          green
38.1    3.178023327  green
15.8    4.550671551  green
31.4    3.823887873  green
43.1    3.5          green
72.1    3.613040829  green
61.3    3.386029412  green
59.9    3.549664839  green
  • 5
    $\begingroup$ Simpson's paradox is essentially a story about omitted variables. In your regression, that variable - group membership - is not missing, so you can just include it in a multiple regression via a dummy variable. $\endgroup$ – Christoph Hanck May 27 '16 at 12:13
  • 1
    $\begingroup$ Ok. That makes sense even in light of my limited knowledge of statistics. So if I were to show that this group membership were strongly predictive of the Y variable, then I could conclude that the membership is perhaps as if not more important than the X variable (depending on their relative predictive strengths)? $\endgroup$ – philosonista May 27 '16 at 12:31
  • 1
    $\begingroup$ Yes, that is what I meant $\endgroup$ – Christoph Hanck May 27 '16 at 12:33
  • $\begingroup$ Brilliantly helpful, Chrtistoph! My team thanks you! $\endgroup$ – philosonista May 27 '16 at 12:35
  • 2
    $\begingroup$ Can you paste the actual numbers in, instead of a picture of them? That way, people will be able to work with your data without trying to laboriously re-type them. $\endgroup$ – gung - Reinstate Monica May 27 '16 at 13:11

You can call it "confounding" or "omitted variable bias":

  1. Confounding: The statistical relationship between $X$ and $Y$ is confounded by the grouping factor "color". [Maybe you remember that a confounder is a variable which is both related to $X$ and $Y$, which is the case in your picture.]

  2. Omitted variable bias: The estimated statistical relationship between $X$ and $Y$ suffers from an omitted variable bias if the grouping factor "color" is omitted.

Both are describing the same problem in different words. Simpson's paradoxon can be viewed as an extreme case of those phenomena.

When I reread your post I was somehow unconfortable about my answer because usually, we want to avoid confounding/omitted variable bias but it seems that you are preferring the "crude", unadjusted relationship between $X$ and $Y$.

Usually, we first describe the crude relationship and then revise it by confounder adjusting it. But I am sure also the converse can be useful in some cases.

Now, how can you "quantify" confounding? In a regression setting, you would compare the estimated slopes between two regressions both run on the pooled data set: One by regressing $Y$ on $X$ and a more complex one by regressing $Y$ on $X$, $\text{Color}$ and eventually on the interaction of $X$ $\text{Color}$ (then you have two comparisons to make). It is a very broad topic, so it is difficult to provide a complete answer.

  • $\begingroup$ Very helpful terminology and thoughts, thank you! Yes, we are partial to the crude relationship, as we're trying to replicate something close to another study that did the same. The crude relationship just turns out to be less appropriate in our study, but the comparison between the two studies is important. $\endgroup$ – philosonista May 27 '16 at 12:36
  • $\begingroup$ @philosonista, if this answer has resolved your issue, you can accept it by clicking the check mark to its left below the vote total. $\endgroup$ – gung - Reinstate Monica May 27 '16 at 13:09

It isn't literally Simpson's paradox, because the definition of that is that the sign of the relationship between X and Y changes upon adding another variable. Since the sign doesn't change, it doesn't quite meet the definition and so, it isn't a case of Simpson's paradox.

However, Simpson's paradox is just a special case of a more general phenomenon. If the third variable, we could call it Z (or in our case color), is not perfectly uncorrelated with X and Y, then the apparent relationship between them will change when we take Z into account. Whether or not the sign flips is dramatic, but not necessarily substantively important. Consider a case where the correlation changes from $r = .1 \rightarrow r = -.1$, and compare that to a case where the correlation changes from $r = .9 \rightarrow r = .1$. In which case might the change have been more important? I suppose it depends on the subject matter, but my first guess would be the latter. For a fuller treatment of this topic, it may help to read my answer here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

At any rate, you want to show that this is what is going on in your data. To establish this, you just have to show that the correlations between color and X, and between color and Y, are not exactly 0 in your sample (you don't need hypothesis tests). You can do that by reporting the two correlations. If you want to make a statement about the values of X and Y differing by color in the population, you could do a couple of t-tests. If you wanted to determine if the relationships between X and Y were the same in both color groups, you would fit a multiple regression model with an interaction and see if the slopes and intercepts were close enough for your purposes to state that the differences 'might as well be 0 for all you care'. (Note that this is an equivalence test, which is an inversion of the more typical logic of hypothesis testing. To understand this more fully, it may help you to read my answer here: Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?)

Here are examples of the analyses I've discussed, coded in R:

## reading in your data:
d = read.table(text="Y       X            Color
               29.2    3.822954823  orange
               59.9    3.549664839  green", header=T)

## showing the correlations aren't 0: 
with(d, cor(Y, Color=="green"))  #  0.4840144
with(d, cor(X, Color=="green"))  # -0.65251

## testing if the groups differ in the population:
t.test(Y~Color, data=d, alternative="greater")
#  Welch Two Sample t-test
# t = 3.009, df = 26.74, p-value = 0.002828
t.test(X~Color, data=d, alternative="less")
#  Welch Two Sample t-test
# t = -4.4478, df = 23.333, p-value = 8.952e-05

## assessing whether the relationship b/t X & Y are similar in the 2 colors:
m = lm(Y~X*Color, d)
# ...
# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)   
# (Intercept)     105.41      35.84   2.941  0.00695 **
# X               -15.07      10.09  -1.493  0.14785   
# Colororange     -61.13      60.89  -1.004  0.32499   
# X:Colororange    13.07      15.51   0.843  0.40739   
# ...

## the linear relationships between X & Y for the two colors are: 
# green:  105.41 - 15.07*X  
# orange:  44.28 -  2.00*X  
## the differences in the intercepts & slopes, w/ 95% CIs are:
difs           = data.frame(coef(m)[3:4], confint(m)[3:4,])
colnames(difs) = c("difference", "2.5 %", "97.5 %")
rownames(difs) = c("intercepts", "slopes")
#            difference      2.5 %   97.5 %
# intercepts  -61.13285 -186.53497 64.26927
# slopes       13.06666  -18.86799 45.00131
  • $\begingroup$ Isn't the regression analysis appear to say that Y does not (significantly) depend on either X or color? That's not the impression one gets from the scatter plot. $\endgroup$ – amoeba says Reinstate Monica May 27 '16 at 18:55
  • $\begingroup$ @amoeba, the regression model has an interaction term. We should be careful of interpreting the main effects. The interaction is not significant. That means there isn't enough evidence to conclude the slopes differ by color. Also, the colors don't differ when X=0, & the slope doesn't differ from 0 in the green color group. To see if the slope differs from 0 overall, we'd need to fit a model w/o the interaction. $\endgroup$ – gung - Reinstate Monica May 27 '16 at 19:16
  • $\begingroup$ So, to follow up: a model w/o the interaction is significant, but neither X nor Color is individually. The VIFs are 1.7. If I test w/ type I SS, p(X) = .009, which is also the p-value (to 3 decimal places) for the simple regression. (Also, a plot of the data looks the same, so I think the data are accurate.) $\endgroup$ – gung - Reinstate Monica May 27 '16 at 19:33
  • $\begingroup$ The impression that I have when looking at the scatter plot is that X and color are quite correlated, and I thought that they both come out non-significant because of that. Which is very misleading, given an obvious effect of either X or color on their own. I would think that the analysis that the OP wanted would be smth like testing Y~color*X against Y~color. Apart from that, I don't think I understood your point about model w/o interaction. If you fit Y~color+X, then color and/or X come out significant? $\endgroup$ – amoeba says Reinstate Monica May 27 '16 at 19:54
  • $\begingroup$ @amoeba, I don't really understand what the issue is here. Y~color is the t-test I do at the beginning. The simple Y~X regression is highly significant, as I mentioned in the last comment. I illustrated different analyses to show how to address different questions (the exact question the OP wants is somewhat ambiguous). The model w/ the interaction allows the OP to test if the slope of the relationship b/t X&Y differs by color. B/c the model has the interaction term, you don't want to interpret the terms / p-values for X or color. You can get that by fitting a model w/o the interaction. $\endgroup$ – gung - Reinstate Monica May 28 '16 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.