Say I have 800 (X,Y) data points, and I do a LSQ fit and get
y = mx+b
Then I think to myself, of the 800 data points, 500 are males and 300 are females, so I should fit them separately. This time I get:
y1 = m1x1 + b1 (males) y2 = m2x2+ b2 (females)
Then I notice that of the 500 males, there are 200 European and 300 American, and of the females, 100 are European, and 200 American, so I divide the data further and do 4 fits:
y3 = m3x3 + b3 (American males) y4 = m4x4+ b4 (American females) y5 = m5x5 + b5 (European males) y6 = m6x6+ b6 (European females)
Then I decide to lump the males and females back together, and just fit the Americans and fit the Europeans:
y7 = m7x7 + b7 (all Americans) y8 = m8x8+ b8 (all Europeans)
Now I'm presented with an out-of-sample Male European. I can use any one of 4 equations to predict his Y value:
y = mx+b y1 = m1x1 + b1 (males) y8 = m8x8+ b8 (Europeans) y5 = m5x5 + b5 (European males)
My prediction 'y5' doesn't have many observations, so will have a larger error due to statistical fluctuations. My prediction 'y' includes all the observations, but will have a larger error due to including unlike observations in the dataset.
If there is in actuality no difference between Americans and Europeans, then dividing them will just introduce errors. If there is a large difference, then it will be important to separate out the data sets.
In reality, choosing one over the other can't be right, because there is information in each fit, so weighting the 4 fits appropriately ought to be better than any one individually.
Is there a way to determine how to weight the 4 predictions based on their expected errors, and come up with a single best prediction?
EDIT: To be clear, I'm looking for a mathematical closed-form algorithm. I'm not looking for a "methodology" like, cross validation, regularization, logistic regression, etc...
There is error introduced because of insufficient data.
There is error introduced because of lumping together unlike observations.
A partition will increase error due to reducing the number of observations in the fit. At the same time, a partition will decrease error to due eliminating unlike observations.