Training sample division vs extra binary/nominal factor Suppose, there is some classification/regression problem. It seems for me, that sometimes division of training sample by some feature (or maybe by some other reasonable method, say feature generation and then devision by it.) and then fitting two models is better then fitting one model. For example, you anylize life duration of people of two contries. There are lots of medical, biological, etc. features of people. you can add one more binary feature that takes value 0 if a man from country_1, and 1 if he is from country_2. What if these contries are "very different"? Intuitively I feel, that fitting models separately for two countries can even improve forecasting quality, despite the fact of lower traing sample. The data coming from one country seems to be more specific and "purer". So, under what conditions devision of training sample is reasonable? What rule of thumbs or explanations in plain English could you advice?
 A: Max, good question.  It sounds like a question of model selection (another tag to add?); you seem to be ultimately asking, "When is Model A (e.g., treating countries separately) better than Model B?"  And you seem to address the issue of misspecification ("if these countries are 'very different'" then Model B would be bad).
I like your example; maybe I can write out some equations for others to help discuss.  My understanding is that $Y$ is lifespan (life duration), $X$ is a vector of "features of people," and we have data from two countries.  Without assuming any further functional form other than independence across people, we could model country 1 with $$Y_i=h_1(X_i,u_i)$$ and country 2 with $$Y_i=h_2(X_i,v_i),$$ where $u_i$ and $v_i$ are error terms.  Alternatively, we can pool the data and write $$Y_i=h_3(X_i,\nu_i)+\gamma c_i,$$ where $\nu_i$ is an error term and $c_i=1$ if individual $i$ is from country 2 and $c_i=0$ otherwise (country 1).
So I think part of your question is, in the "real world," is the actual predicted lifespan difference between the two countries just a constant ($\gamma$), or does it depend on $X_i$?  And the other part is, will a forecast based on a more flexible model actually be better?  I think knowledge of the actual phenomenon being studied (e.g., lifespan) is irreplaceable, but there are also many existing (automated) ways to choose among different models (or combine them by "model averaging"), such as Aikake's information criterion (AIC), the Schwarz/Bayesian information criterion (BIC, or SIC or SBIC), cross-validation (CV), etc.--the text by Claeskens and Hjort is a good one, but maybe other people have other suggestions.
I think the basic idea of model selection is, we know that (in this case) the model with the dummy variable is misspecified, but if it's "pretty close" then we might prefer it to a more flexible model that has significantly higher variance.  The usual mean squared error (MSE) measure of an estimator's accuracy is the variance plus the square of the bias--it can be better to be a little biased if you can lower the variance a lot.
In the case of the lifespan model, I would guess researchers use something in between, depending on the mix of countries.  (E.g., if you have Norway and Sweden, you should model it differently than Switzerland and Malawi.)  Ideally, we can understand real world phenomenon that work similarly even in different countries, so ideally we would include all countries in the same estimation.  And of course, should always be careful of the difference between forecasting/prediction and understanding causality, but that is another topic.
Hope something in there helps, or can lay some groundwork for other answers!
Dave
