Say I commit the following sins while building a predictive model:

  1. I take my dataset and split it into four subsets: Three for training
    (Train_A, Train_B, and Train_C) and one for validation.

  2. I train an initial model (Model_A) on Train_A. Because the goal is to maximize out-of-sample prediction accuracy, I use bias-variance-balancing techniques like cross-validation.

  3. I generate predictions from Model_A on Train_B and record the prediction errors.

  4. Next, I train a second model (Model_B) on Train_B, but I weigh the observations based on the magnitude of the prediction errors from Model_A. In other words, Model_B is told to focus most on learning to predict the observations that Model_A was really bad at predicting. Again, the goal is out-of-sample accuracy, so a technique like cross-validation is used.

  5. I generate predictions from Model_A and Model_B on Train_C. These are used to explore the best way to combine the predictions (e.g., weighted average) from both models to (hopefully) increase out-of-sample prediction accuracy.

  6. After determining the best way to weigh the predictions from Model_A and Model_B, I estimate the out-of-sample accuracy using the validation set.

Main Question: Am I damned? Is this approach inherently and irrecoverably prone to overfitting? Or, is there a way to use the errors from Model_A to inform how Model_B is trained in such a way that the strengths of Model_B address the weaknesses of Model_A?

Secondary Questions: Are there particular techniques or algorithms that are better/worse at extracting value from this kind of approach? For example, I wouldn't be surprised if there are some NN techniques that inherently do this kind of thing and, therefore, wouldn't benefit at all from this approach whereas something less flexible (like regularized regression) could potentially benefit greatly in comparison. What other thoughts or advice would you provide to someone who wishes to take this approach?

Thank you!

[Edit: I feel like I walked into a McMenamins and pitched the idea of a Microbrewery to the bartender, haha! Thanks everyone for your very kind and helpful comments!]

  • 9
    $\begingroup$ See boosting algorithms. They do more or less what you describe. Typically, see AdaBoost fr.wikipedia.org/wiki/AdaBoost . $\endgroup$
    – TMat
    May 5, 2021 at 14:59
  • $\begingroup$ Thanks! My assumption was that I couldn't possibly be the first to think of this. However, I am curious if there are any red flags in the approach I've outlined. It is nice to know that there are algorithms that already do this, but can the approach I described above be manually applied to any algorithm? $\endgroup$
    – Jdclark
    May 5, 2021 at 15:03
  • 1
    $\begingroup$ Geez I feel pretty silly now after looking into boosting. Turns out the basic answer to my question wasn't easily found here because the answer is so obvious lol. Still, I'm curious if I there are any special considerations I should make if I were to take the approach manually. I see there are algorithms that already do this, but do you see any red flags in my THINKING? $\endgroup$
    – Jdclark
    May 5, 2021 at 15:16
  • 1
    $\begingroup$ What you call validation, I would call testing, while what you are doing taking a model from TrainA and predicting TrainB etc. and then adjusting and tuning the model is what I would call validation. So be aware when you read other people, they may use slightly different language while doing similar things $\endgroup$
    – Henry
    May 5, 2021 at 15:24
  • 2
    $\begingroup$ In addition to all the responses about boosted tree models (such as xgboost, LightGBM etc.), which admittedly don't work with iterative sets of of datasets, there is indeed a neural network that tries to marry boosting on residual errors: TabNet (arxiv.org/abs/1908.07442). $\endgroup$
    – Björn
    May 5, 2021 at 16:29

2 Answers 2


As noticed in the comments, you’ve re-discovered boosting. Nothing wrong with this approach, but usually it’s easier and safer to use a method already implemented and battle-tested by someone else than starting from scratch. If you really want to use your approach, I’d encourage you to first use some out-of-the-box implementation of boosting (AdaBoost, XGBoost, CatBoost, etc) to use it as a benchmark.

  • 3
    $\begingroup$ (+1) This also gives you some idea of the performance you can hope to expect with your own method. $\endgroup$ May 5, 2021 at 15:43
  • $\begingroup$ Thanks so much for your help! Of course, I learned about "boosting" in school but never engaged with it enough to remember it. So I SHOULD know this and I'm a little embarrassed. Still, I have hope that some people reading this will get a good laugh. If so, worth it. $\endgroup$
    – Jdclark
    May 5, 2021 at 15:56
  • 9
    $\begingroup$ @Jdclark people are re-discovering stuff all the time, everyone does that. Sometimes it’s hard to notice by yourself what you didn’t think about, or recognize. $\endgroup$
    – Tim
    May 5, 2021 at 16:04

As was mentioned in the comments this idea of iteratively learning from previous model errors is at the core of boosting methodologies like Adaboost or gradient boosting.

As you theorize the idea is prone to overfitting in certain models like trees but it actually regularizes a model such a linear regression (although I would just do standard regularization like l2 normally). In terms of algorithms which do well with this typically it's trees (xgboost or lightgbm are go-tos for hammers in the data science community) or some approach which partitions your data. This is because each time you refit the model you get new splits and the tree can learn new things whereas in linear regression it just updates your coefficients so you aren't actually adding any complexity or anything. Adding two regression models just averages the coefficients but adding two tree models gives you a new tree.

This is similar to bagging predictors, bagging linear models will converge to fitting on the whole set whereas bagging trees actually benefits you in terms of the bias-variance tradeoff.

In terms of NNs, I believe there is some theory connecting gradient boosting to residual networks and similar architectures see this question on it.

My recommendation is just use lightgbm or xgboost!

  • $\begingroup$ Thank you so much! I especially appreciate the comparison between how boosting affects tree vs. regression models. $\endgroup$
    – Jdclark
    May 5, 2021 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.