Interpreting sub-sample analysis when coefficient signs are opposite

Question

Suppose I run the following OLS regression:

$y = \beta_0 + \beta_1 D + \varepsilon$

where $D$ is a binary variable for treatment group, and $y$ is the outcome variable, say income. $D$ is randomly assigned to the sample, so $\beta_1$ represent causal effect of the treatment on $y$.

Suppose $\beta_1$ is positive and statistically significant. Now the researcher wants to split the sample into 4 (or any integer n > 1) sub-sample. Let's denote the subsample by $S$. Let's say researcher divided the sample by quartile of age. Researcher runs the same regression as above, and get every $\beta_1^S$ coefficients in the subsample to be negative and statistically significant.

How should researcher interpret the results? I understand that this can happen by chance, depending on how the data looks like, like this post. However, I'm not sure how to interpret the results, especially when I'm making causal statement.

Do I say that treatment $D$ causes $y$ to increase by $\beta_1$ for the overall sample, but when looking at subsample, $D$ causes $y$ to decrease by $\beta_1^S$ for subsample $S$? It's difficult for me wrap my head around the interpretation of the results.

Answer 1

First, I would not divide by quartile of age. If you are interested in age, I'd add it (in years) to the model, maybe add a spline of it, and maybe add interactions with D.

Second, I would not try to interpret the coefficients without looking at a graph. Graphics are not some separate thing, they are a key part of analysis. They may point out something fascinating, they may show a violation of assumptions, etc. This is especially true when results are "odd" in some way (as yours are). Could I try to figure out what these results mean, without a graph? Sure. I might even come up with invented data that did this. But .... It would be hard. And, if it would be hard for me to do it, then it would be even harder for whoever I am explaining it to to understand (given that I have a PhD in psychometrics and have been doing data analysis for a few decades). USE A GRAPH. I am pretty sure that, in a case like this, I would not only want the graph for me, but would include it in any write-up or presentation.

Answer 2

The interpretation depends strongly on exactly what your actual variables are and how your study was designed.

In my experience, Simpson's Paradox usually happens with observational data, where the "treatment" isn't actually randomly assigned, and some subgroups have many more "treatment" units while other subgroups have many more "control" units. In that case, the subgroup-by-subgroup interpretation is often more sensible, and the combined-data results can often be ignored.

On the other hand, if your treatment was randomly assigned, and if the grouping variable was used for blocking as part of the study design, then the number of treatments and controls should be the same within each subgroup. If so, intuition suggests you shouldn't see this kind of Simpson's Paradox behavior. Could there be a mistake in your analysis? (If you really do see this effect in a controlled experiment with blocking, I'd love to hear more details & try to understand how it happened.)

That doesn't sound quite like your hypothetical example, though. Your subgroups are age groups, which should have approximately the same distributions in the treatment and control groups, but not necessarily exactly the same (if you didn't block on age when you assigned treatments). So maybe double-check whether your age groups are so small that their distributions differ a lot between treatment and control AND whether the difference in age mixes could be giving you your Simpson's Paradox results.

Finally, even if the treatment is randomly assigned, another way to get a Simpson's Paradox situation is if the subgrouping variable is on the causal pathway. There's an example in Section 1.2 of Causal Inference in Statistics: A Primer by Pearl, Glymour, and Jewell. In their case, there is random assignment to Drug or Control (No Drug), and the subgroup variable is post-treatment blood pressure. Their hypothetical drug works by lowering blood pressure, so even though the treatment assignment is random, you still get very different sizes of low-BP vs high-BP subgroups in the Drug vs the Control groups, which can lead to a Simpson's Paradox. In this scenario, they suggest interpreting the combined-data results, and ignoring the breakdown by subgroups---this is just not a useful subgrouping variable. That doesn't sound like your example, but it's hard to tell without knowing your real context for certain.

Answer 3

In an analysis of variance class I explained this with tables, one containing cell means of the dependent and one containing the corresponding cell frequencies:

For each of the four Gr groups the treatment "causes" a decrease of 1 point in the mean of the dependent. However, due to the unbalanced frequencies, the overall Treatment mean is 0.5 higher than the Control mean and thus opposite to what is seen within each of the four Gr(oups).

With balanced frequencies, say each cell containing n=4 (1/4 of the total per row), the overall Treatment mean 2.5 would lie exactly in the middle of the four Treatment means, and the overall Control mean would be 3.5. Hence, the overall difference between Treatment and Control would then be equal to the within GR1, GR2, etc. difference, namely 1 point.

This example is nothing new of course, just showing in a different way what was already said in the earlier comments. The question to be answered is: is it meaningful to control for Gr? Looking per Gr group, as the questions says, is (more or less) similar to controlling for Gr in a regression or ANOVA model. In an ANOVA the four Gr(oups) could act as a second factor; the interaction Treatment*Gr would be zero in the example. Also, the estimated marginal means (EMM) of Treatment and Control would be equal to 2.5 and 3.5, neutralizing the influences of the different frequencies on the marginal means.

So, is controlling for Gr necessary or desired? If there is some causal influence of Gr on the dependent, than the answer is 'yes', I would say.

However ... the nice answer of Civilstat about mediation by the Gr(oups) offers a different view. I will try to explain such mediation for my example data.

Suppose that the composition of the four Gr(oups) is influenced by the treatment. It could be that the treatment causes people to more "prefer" Gr4 instead of Gr1. Prefer may not be the right word here, as there may be all kinds of reasons why people more belong to Gr4 after the treatment. The point is that the treatment causes the "move" away from Gr1 and into Gr4. So, this change is not coincidental or manipulated by the researcher, but it is a truly causal influence of the treatment. We then have the following causal chain:

Instead of using the entire variable Gr as the mediating variable, in the above diagram I used only Gr4, or the proportion of people belonging to Gr4. This is explained now. The Gr(oup) composition changes as a consequence of the treatment, but only for the proportions of people in Gr1 and Gr4. The proportion of people in Gr4 is 2/16 for Control and 10/16 for Treatment, so a shift of +8/16. We can also say that the proportion in Gr1 changes by -8/16. If the proportions in Gr2 and Gr3 do NOT change, then the (absolute) proportional shift away from Gr1 goes together with the same proportional shift into Gr4. In the chain therefore only Gr4 is given as mediator, with +8/16 as the regression effect of the Treatment on Gr4. The regression effect of Gr4 on the dependent is +3, because we only need to look at the move from Gr1 into Gr4 and these groups differ +3 in there means, as shown within each row of the upper left table containing means. In total then the mediated or indirect influence from Treatment on the Dependent can be calculated as 8/16 * 3 = +1.5. The direct effect of Treatment on the dependent is -1 as is also shown, within columns, in the upper left table and also along the curved arrow in the causal diagram. The total influence of Treatment now is equal to the sum of the direct and indirect effect of Treatment, -1 + 1.5 = +0.5.

Another, may be simpler, method to calculate the indirect influence of Treatment on the dependent is as follows. Going from Control to Treatment the Gr(oups) composition changes. We could now ask: how does Gr composition alone affect the dependent, i.e. when keeping Treatment constant or when controlling for Treatment? Knowing this change in the dependent shows us the influence of Treatment on the dependent as far as it is caused by the changing Gr(oups) composition only. Or: it shows us the indirect influence via Gr composition. Suppose we apply the Gr frequencies (2, 2, 2, 10) for Treatment to the data of the Control group ... this would produce a mean of (22 + 23 + 24 + 105) / 16 = 68/16 = 4.25. So, this different Gr composition would produces a mean of 4.25, whereas with Gr composition (10, 2, 2, 2) the Control mean was 2.75 (see table). That is, the Gr composition shift alone leads to a increase of 4.25 - 2.75 = +1.5; this 1.5 increase is purely caused by the shift in composition of the four Gr groups, which is one of the things that change when comparing Control and Treatment! So here we (again) have the indirect or mediated influence of Treatment.

And finally, the question at issue again is: which influence to report, -1 or +0.5? In case of mediation, both would be relevant. And if the Gr(oups) frequencies are in no way caused by the Treatment, and the Gr(oups) have an indisputable influence on the dependent, it seems more plausible to me to consider -1 as the "true" Treatment effect.

Answer 4

Many of the answers here are quite good, but I wanted to add on one more comment that might clear up some confusion: the example that you've given would lead to an outcome that is incredibly difficult to interpret and is non-sensical. To illustrate how wild an example you've come up with, consider what would happen when you rank the outcomes by age of the patient: you would have a spike at the beginning a each quartile, despite the fact that the quartiles are arbitrary barriers. That means if patients in your study were from the age of 20 - 59 you would find the treatment most effective at 20, 30, 40, and 50 and least effective at 29, 39, 49, and 59. Nonsense.

This highlights a key fact: continuous data makes for terrible grouping variables. In the example you've given, someone who is 30 is completely different from someone who is 29, which doesn't make any sense.

For an easier to understand example, consider instead that you are looking at an animal's lifespan based off its body size, and you have three types of animals: mice, dogs, and elephants. You will find an overall trend that as these animals get bigger, they will live for longer, but, within each species it is the smaller individuals that live longer. So a dog will live longer than a mouse, but a small dog will live longer than a big dog. This is an overall positive effect of body size on life span, but a negative effect within each group.

Now, potentially moving outside the scope of your original question: you do not talk about whether your additional variable is an interaction or a random effect - this is very important for the interpretation of the outcome. If you fit my example above as an interaction, you will find a positive effect of body size on lifespan, but a negative interaction between each species and body size. If you fit species as a random effect, you will find only a negative effect of body size, as you have allowed the intercept for each species to vary and you are only looking at within-group trends. When you decide to add something as a fixed versus random effect, consider the question that you are trying to ask and what the model actually tells you.