0
$\begingroup$

Say I have 800 (X,Y) data points, and I do a LSQ fit and get

y = mx+b

Then I think to myself, of the 800 data points, 500 are males and 300 are females, so I should fit them separately. This time I get:

y1 = m1x1 + b1 (males)
y2 = m2x2+ b2 (females)

Then I notice that of the 500 males, there are 200 European and 300 American, and of the females, 100 are European, and 200 American, so I divide the data further and do 4 fits:

y3 = m3x3 + b3 (American males)
y4 = m4x4+ b4 (American females)
y5 = m5x5 + b5 (European males)
y6 = m6x6+ b6 (European females)

Then I decide to lump the males and females back together, and just fit the Americans and fit the Europeans:

y7 = m7x7 + b7 (all Americans)
y8 = m8x8+ b8 (all Europeans)

Now I'm presented with an out-of-sample Male European. I can use any one of 4 equations to predict his Y value:

y = mx+b
y1 = m1x1 + b1 (males)
y8 = m8x8+ b8 (Europeans)
y5 = m5x5 + b5 (European males)

My prediction 'y5' doesn't have many observations, so will have a larger error due to statistical fluctuations. My prediction 'y' includes all the observations, but will have a larger error due to including unlike observations in the dataset.

If there is in actuality no difference between Americans and Europeans, then dividing them will just introduce errors. If there is a large difference, then it will be important to separate out the data sets.

In reality, choosing one over the other can't be right, because there is information in each fit, so weighting the 4 fits appropriately ought to be better than any one individually.

Is there a way to determine how to weight the 4 predictions based on their expected errors, and come up with a single best prediction?

EDIT: To be clear, I'm looking for a mathematical closed-form algorithm. I'm not looking for a "methodology" like, cross validation, regularization, logistic regression, etc...

There is error introduced because of insufficient data.
There is error introduced because of lumping together unlike observations.

A partition will increase error due to reducing the number of observations in the fit. At the same time, a partition will decrease error to due eliminating unlike observations.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Rather than constantly subdivide your data, why not simply fit a single multiple regression model with each of the variables that are likely to be important in the model?

If you had a 1000 observations and ten such grouping variables like sex that roughly split the data in half, on average each regression would have less than one observation -- yet 1000 observations with ten predictors (or eleven if you count the x-variable) all in one equation may be quite feasible.

With some regularization you might reasonably incorporate all the two-way interactions you felt worth including. (On the other hand if you need all higher way interactions, you would be back in the same boat you started with, and then regularization - or some way to reduce the space of possible models - would be necessary.)

Improve this answer
$\endgroup$
6
  • $\begingroup$ The point of subdividing and then doing several multiple regressions is to let the data tell me everything it knows. As you say, I might "reasonably incorporate all the two-way interactions I felt worth including", but rather than relying on my "feelings", the data should speak for itself. I don't want to hand-pick the partitions. If I use every possible combination of partitions, the ones which use every partition will, as you say, become meaningless because of the sparsity, but the data can tell me that. The data should be able to tell me how many and which partitions are best. $\endgroup$
    – TPM
    Dec 13, 2015 at 14:24
  • $\begingroup$ The problem is you don't typically have the data to incorporate all possible interactions when there's more than a handful of variables (which would usually imply a designed experiment set up to eliminate other sources of variation); you'd need to apply some form of regularization, which is already covered in my answer. If you're also interested in selecting a model (as your comment suggests) you'll also need some form of sample-splitting/cross-validation, which makes the observation shortfall worse, so the need for regularization is stronger still. $\endgroup$
    – Glen_b
    Dec 13, 2015 at 22:42
  • $\begingroup$ Hi Glen, thanks for trying. I admit it is a very hard problem. I do have quite a bit of data, so the "typical case" that you mention isn't relevant. I agree with you that cross-validation is clearly not the way to go, and regularization doesn't address the question either. I think you are on the right track with "reducing the space of possible models". I'm pretty sure the data can tell us how to do that, rather than an ad hoc approach, I'm just not sure how to go about it. $\endgroup$
    – TPM
    Dec 14, 2015 at 1:50
  • $\begingroup$ @TPM I don't see how you could construe my comment as suggesting that "cross-validation is clearly not the way to go". ... if anything I was saying it was a good idea if you're trying to both select and estimate a model (since you don't want to do them on the same data). If you have a lot of data, however, I am not sure why there's a difficulty; estimate all the interactions you like, but have some regularization to help deal with the areas where there's no a lot of data. (This might include mixed models, for example.) $\endgroup$
    – Glen_b
    Dec 14, 2015 at 1:55
  • $\begingroup$ As a more concrete example, if we had 10 trillion observations, we'd surely be happy breaking the fits out my male/female, American/European. If instead we had 10 observations, we wouldn't be able to partition the data at all. Somewhere between 10 and 10 trillion, we'd decide we had enough data to split it up. And where that point is will be a function of how well the conditions separate out the data. It could be that at 4,000 observations, the error we create by splitting by Male is less than the error created by not splitting by male. That's the relationship I'm looking for. $\endgroup$
    – TPM
    Dec 14, 2015 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.