Negative correlation in two groups (separately) but overall data shows positive correlation Regression on the overall data gives positive correlation coefficient. However, if I divide the data by gender (male and female) and run the same regression model separately on each group - I get negative correlation. This is happening maybe because the there is a big difference between the genders and my overall regression is not able to interpret that.
I want to incorporate this effect into the model. How do I do it?
I tried just adding a binary variable gender to the model but it doesn't change the results much.
 A: There are many ways your regression can be confounded by other aspects. It is not surprising your data can have such radical differences if there are other confounds in your data that are not accounted for.
Using the iris dataset in R as an example, here is a regression between the dimensions for sepals from flowers:
iris %>% 
  ggplot(aes(x=Sepal.Width,
             y=Sepal.Length))+
  geom_point()+
  geom_smooth(method = "lm")


You can see the relationship is weak and negative. The data points look fairly odd in their distribution too. However, if we split the regressions up by species:
iris %>% 
  ggplot(aes(x=Sepal.Width,
             y=Sepal.Length,
             color=Species))+
  geom_point()+
  geom_smooth(method = "lm")

We see something totally different: positive and stronger relationships between sepal dimensions, and they are more linear given the data points are dispersed primarily because of species dimensions. You can also see the standard error for prediction (the gray area) has gone down considerably for the setosa values:

