Pool samples which "look" too different? Let's say I am trying to estimate $E(y | x_1, x_2,...)$. The dataset has two subcategories, say URBAN and RURAL. The descriptive stats of the two subsets are very different, as are the coefficients from regressions run separately. For example, I can see that $E(y|x_1)$, $E(y|x_2)$ etc. are all positive for URBAN and all negative for rural. In this case, is it still justified to run a pooled regression with interactions thrown in? Or, do we assume that two samples are really from two different populations and hence the models should be estimated separately? Is there any way to test (frequentist or bayesian) if the two samples came from the same underlying population?
 A: *

*In this case, is it still justified to run a pooled regression with interactions thrown in? Or, do we assume that two samples are really from two different populations and hence the models should be estimated separately? 


The key is to apply the same causal modeling framework that rationalized the model without consideration for URBAN/RURAL designation, but then apply rurality as a causal factor. For instance, you know that rural people have less access to health care, so if it is a study of prevalence of disease versus, say, family history of disease, you may choose to adjust for rurality as a confounder. If you adjust for a model controlling for rurality and its interaction with all other model effects, you obtain inference that is equivalent to just fitting two separate models for rural and urban observations. Systematically testing interactions in a stepwise fashion does not make any sense here.


*Is there any way to test (frequentist or bayesian) if the two samples came from the same underlying population?


No, this is a common theme and it is a rabbit hole. It is the scientist's notion of how the samples were collected that should determine whether they are okay to be "pooled" in any sense. You will tend to rule that samples are different more often than when they are the same. Calibrating "homogeneity" tests is a nightmare. Plus once you make a decision, you would need some very complicated way to account for your structural testing to obtain the "right" p-values... because when you test a bunch of things, then decide on a model, that model's default p-value does not represent the right p-value. This is a multiplicity issue.
If it is any reassurance, it will not be incorrect in any way to estimate two separate models except for a possible hit to power.
A: This depends on what is know of the domain, that is the origin of your data. As the data is already labelled URBAN or RURAL, it seems the hypothesis is there are two populations.
Then, you say the descriptive stats are different, including correlation coefficients having a different sign. Based on this i assume that difference is so obvious that mathematical tests are unneccesary. These could otherwise include a t-test to check if there is a significant difference between the means of each $x_i$ for the two groups, or the Fisher transformation to compare correlations.
Estimate the models seperately. Mistakenly pooling groups will lead to incorrect models, in your case (coefficients with opposing sign) effects could cancel each other out. See also Simpson's Paradox
