Predictive model for error of another model

Question

Does it make any sense to build a regression model for a certain target variable on a certain training set. Then build a regression model for the errors of the previous model ( real values vs predicted). And then add the results of both models on a test set. Has anyone heard of anything like this? I tried to do some prior research but dont know how to even begin searching for this.

To explain a little bit more, I'm trying to do an interpolation of maximum tree heights per squared KM over a certain territory. I tried with a random forest regression wich does quite well but I wanted to improve it bit if there is a chance. I have big errors (this is all on a 20% test set) when I try to predict very tall trees or very short trees. The model is biased and doesnt have enough information to "see" these big and small heights. I was looking for a way to correct this if theres is one. I uploaded an image of what is happening, I ordered the error from smallest (overestimation) to largest (subestimation). I hope it helps.

Answer 1

There are certain situations where this makes sense. In fact, what you describe is a simple form of boosting or stacking.

One situation where this might make sense to do manually would be if you had the output from several different regression models, and wanted to combine them together in an ensemble. You could feed the outputs from your primary models into a secondary regression, which would determine how to optimally weight them together.

Note that this procedure would require 2 test sets (or better yet 2 rounds of cross-validation). You would use the first test set to determine the weights for the secondary regression, and the second test set to determine the generalization error of your full model.

Answer 2

I don't completely understand your question, and anyways we cannot possibly say whether this is sensible if you don't tell us what you model (application) and how.

However, recognize two situations where building a model that describes errors does make sense

• hyperparameter optimization (e.g. cost and γ for SVM). The performance/error of the SVM could be modeled as function of the hyperparameters.
  (the hyperparameters would be @whuber's withheld variables).

• In analytical chemistry, the test set performance is often modeled pred. conc. = lm (true conc.). The slope (aka recovery) can be $\neq$ 1, e.g. if part of your analyte (= substance you want to measure) is masked by the sample matrix (= other stuff that makes up the physical specimen).
  Here's a hypothetical example, where 10% of the analyte are not detected (recovery rate 90%) :
  
  The predicted concentrations should be around the gray line (where $c_{predicted} = c_{reference~method}$). The predictions however, are systematically too low ($c_{predicted}$ below gray line), and the bias depends on the true concentration $c_{reference~method}$. A linear model regressing the predictions against the reference values models these systematic deviations (dashed line).

Literature: Danzer, Analytical Chemistry: Theoretical and Metrological Fundamentals