Combining models for prediction based on residual performance

Question

I have never read or seen someone do this before, so I wanted to pose the question here. Suppose I fit a basic linear model, $\text{price of house} = \beta_0 + \beta_1*\text{taxes} + \beta_2*\text{bedrooms} + \beta_3*\text{number of garage doors}$.

I fit this using OLS and I get a model that has combined $R^2$ of say .68... and I look at the residuals from the actuals vs predicted plot, and I see that the model does pretty well for houses that have less than 3 bedrooms, and maybe more than 2 garage doors....

I then build a robust linear regression model, or maybe a neural net, and I get a model that now performs worse as a whole, with a combined $R^2$ of say .53 BUT when I look at the residuals from the actuals vs predicted plot, I see that this model, model #2 was actually better a predicting the sales price for houses that model #1 did a poor job of predicting, say the outliers.... this scenario goes on and on, and lets say I have identified 5 different models, that for different parts of the population, do a better job of predicting those parts of the populations and these parts are not mutually exclusive.

Is there are known approaches or good documentation on say using diagnostics to identify where your model may be poorly performing (on what types of data), and then some standardized approach to say joining predictions from multiple models to get the highest $R^2$? I am not suggesting averaging model predictions, or what is generally seen in ensembles, at least in how I have seen them defined. I am saying a tactic to apply different types of models to different parts of a dataset, (through semi-automatic identifcation of which model is best applied where), and then joining up all of the best predictions.

I do not think segmenting the population first, and then building seperate models for each segment of the population is the best answer here, but is certainly one way to pseudo tackle it.

Hopefully this concept makes sense, as I think it would be very powerful. Thanks to everyone in advance for providing your thoughts and or supporting documentation / literature.

Answer 1

I've used a competitive mixture of experts ensemble successfully in the past to solve this type of problem. In this formulation, the key is that you are training the partitioning/gating function (typically the softmax function) and the underlying experts (linear regressions) at the same time. In this manner, the input data windows that are passed to a particular expert have input attributes that are in similar portions of attribute space but also share a similar relationship to the truth output. Typical mixture of experts papers employ a cooperative framework, in which the output is the sum of the regression output of each expert multiplied by the probability that the expert is responsible for the window of data (with the probability determined by the output of the softmax gating network). In a competitive framework, it is just winner take all; the output is just the regression output of the expert that had the highest probability.

Relevant papers:

Adaptive mixtures of local experts

Hierarchical mixtures of experts and the EM algorithm

Answer 2

I suggest you to look at hedonic regression before getting into this subject. This type of regression is a standard in pricing stuff like houses.

Once, you are done with the above, look up the literature on "combining forecasts". Yes, sometimes simple median of different forecasts works great. I'm afraid not in your case though. You have all the variables and model specifications under control, so it must be better to build them all into one model.

Lastly, sometimes you end up building separate models for different enough subproducts. For instance, it is common to model "subprime" and "prime" mortgages completely separately. Basically, they'd be estimated on different data sets and the models would have different specifications. One can argue whether it's the best approach, but the fact that people do it often.