Negative correlation in two groups (separately) but overall data shows positive correlation

Regression on the overall data gives positive correlation coefficient. However, if I divide the data by gender (male and female) and run the same regression model separately on each group - I get negative correlation. This is happening maybe because the there is a big difference between the genders and my overall regression is not able to interpret that.

I want to incorporate this effect into the model. How do I do it?

I tried just adding a binary variable gender to the model but it doesn't change the results much.

• Have you plotted the data? Is Simpson's paradox possible here? Commented Sep 6, 2022 at 2:01
• @ Galen, Yes. Looks like it is. I didn't know what it's called. Thanks, do you have any suggested reading book/articles on how to resolve this issue? Commented Sep 6, 2022 at 13:12
• What question are you trying to answer with this model? Commented Sep 6, 2022 at 14:04
• I am trying to build a predictive model and there seems to be a distinct difference between two groups within the data. Commented Sep 6, 2022 at 15:48
• A purely predictive goal makes handling Simpson's paradox very straightforward: use whichever way gave you the best testing error. If you are concerned with causality, then I would recommend Causal Inference in Statistics: A Primer by Pearl, Glymour, and Jewell. Commented Sep 6, 2022 at 16:17

There are many ways your regression can be confounded by other aspects. It is not surprising your data can have such radical differences if there are other confounds in your data that are not accounted for.

Using the iris dataset in R as an example, here is a regression between the dimensions for sepals from flowers:

iris %>%
ggplot(aes(x=Sepal.Width,
y=Sepal.Length))+
geom_point()+
geom_smooth(method = "lm")

You can see the relationship is weak and negative. The data points look fairly odd in their distribution too. However, if we split the regressions up by species:

iris %>%
ggplot(aes(x=Sepal.Width,
y=Sepal.Length,
color=Species))+
geom_point()+
geom_smooth(method = "lm")

We see something totally different: positive and stronger relationships between sepal dimensions, and they are more linear given the data points are dispersed primarily because of species dimensions. You can also see the standard error for prediction (the gray area) has gone down considerably for the setosa values:

• Thank you so much for this elaborate answer. This is really helpful. Do you have any suggested book/articles for this? Maybe there are ways to incorporate the "species" in the model without separating the data. My data does exactly something like this! Commented Sep 6, 2022 at 13:02
• Regression and Other Stories. A free and worthwhile read. Commented Sep 6, 2022 at 14:03
• @mgdata If you find an answer useful, please don't forget to upvote it. Commented Sep 6, 2022 at 16:31
• Yes. I tried to but it is not letting me upvote because i am new and I don't have '15 reputation'! :( Commented Sep 8, 2022 at 16:34