I'm stacking several models to improve a regression task. My question regards making final predictions when making use of stacking. My set-up is as follows.
I have a train and a test data set.
For the first level I divide the train data set in 6 parts (1 .. 6). I then use parts 2-5 as training data and part 6 as validation data for a model, say a NN. I can now predict part 1 using the resulting NN safely and use it as input for a second level model. I can get estimates for my complete training data set by rotating this set-up. I estimate a single set of hyper parameters for all rotations using the validation parts only. Repeating this setup for other models (random forrest, xgboost) I can then use a stacking model (a linear model for example) to make the final prediction. I could use a hold-out set to assess the quality of the stacked outcomes.
For making a prediction of the test data, the most straight forward approach would be to take the hyper parameters of a first level model, apply those to model and train the model it on all the data. After training I can then use the second level model to aggregate these into a test prediction.
The case in point is that one of the first level models is a NN. Since NN's are very sensitive to starting values I do not want to take the hyper parameters and train a new NN on the complete train data. Instead, to make sure I get a foreseen result with the test data, I average the predictions of the models of the six folds I and use this mean as input to the second level model.
This procedure seems to me a lot like bagging, and in that sense should not negatively influence the end result. Are there reasons to assume the contrary?
