I'm stacking several models to improve a regression task. My question regards making final predictions when making use of stacking. My set-up is as follows.

I have a train and a test data set.

For the first level I divide the train data set in 6 parts (1 .. 6). I then use parts 2-5 as training data and part 6 as validation data for a model, say a NN. I can now predict part 1 using the resulting NN safely and use it as input for a second level model. I can get estimates for my complete training data set by rotating this set-up. I estimate a single set of hyper parameters for all rotations using the validation parts only. Repeating this setup for other models (random forrest, xgboost) I can then use a stacking model (a linear model for example) to make the final prediction. I could use a hold-out set to assess the quality of the stacked outcomes.

For making a prediction of the test data, the most straight forward approach would be to take the hyper parameters of a first level model, apply those to model and train the model it on all the data. After training I can then use the second level model to aggregate these into a test prediction.

The case in point is that one of the first level models is a NN. Since NN's are very sensitive to starting values I do not want to take the hyper parameters and train a new NN on the complete train data. Instead, to make sure I get a foreseen result with the test data, I average the predictions of the models of the six folds I and use this mean as input to the second level model.

This procedure seems to me a lot like bagging, and in that sense should not negatively influence the end result. Are there reasons to assume the contrary?


1 Answer 1


Answering my own question. As per http://ciml.info/dl/v0_8/ciml-v0_8-ch04.pdf averaging across folds is standard practice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.