Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for training of combiner of predictions (layer-2, right figure).

In my project, I have two different multi-classification models. And I have a dataset (train/dev/test) which was used for training and testing two models. When I have learned Stacking, I thought I tried to use the whole training set for the blending training set (layer-2), then, test the blender with the test data. Though I read the book and other websites, and they mention the training set is splitted into subsets.

Is it uncommon (or not recommended) to use the whole training set for both layer-1 and 2? I thought this is not wrong since test data has been already prepared. I have already trained my model by the whole training dataset. So if it is not recommended, should I train my models with splitted training dataset?

The images are cited from "Hands on Machine Learning with scikit-learn and Tensorflow." (2017).

You have to have real, out-of-sample, predictions as the input to your blender, otherwise your blender is not learning about, and thereby improving, prediction accuracy - but instead learning about, and thereby improving, in-sample estimation accuracy, which can lead to overfitting. This is why you cannot use the whole training set for both layers - if you do, some of the "predictions" made by the base models will actually be in-sample estimates, not out-of-sample predictions.

You split your training data set so that subset 1 is used to train your base models. This is what is shown in your left picture. Your base models then are used to generate predictions for subset 2, and these predictions, along with the actuals for subset 2, are given to your blender for training. This is what is shown in your right picture. Basically, the predictions are features that are given to your blender, along possibly with other features from subset 2.

The model that the blender comes up with based on subset 2 is then used to predict the test data. This can be done by predicting the test data with each of the base models (developed on subset 1), then predicting the test data again with the blender model (developed on subset 2) + the predictions from the base models. The resultant predictions are the ones you use for calibrating / testing the combined base models + blender model.

Alternatively, you can re-train your base models on subset 1 and 2 prior to making predictions for the test data set. This will tend to improve the base model predictions of the test data set, but (hopefully slightly) weaken the link between the base models and the blender model, as the blender saw less-accurate predictions when it was being trained. The blender will consequently add less value and more overfitting, but given that the base models are more accurate, it may balance out.

ETA (from comments): In practice, I tend to split into more than 2 groups, liking 10 groups for some reason. The base models are then trained with much more data so are more accurate (at least in situations where you don't have overwhelming amounts of data) and the blender is trained on predictions from models that have accuracy characteristics that are closer to what it will see when going operational, which is a win-win in accuracy terms.

• The models are not outputting predictions for the training set, they are outputting estimates for the training set. The difference is key: predictions are out-of-sample, estimates are in-sample. If your objective is to improve predictive power, you need to train the blender on predictions and actual values, not on in-sample estimates and actual values. Just evaluating the blender on the test data set doesn't change the fact that it was fitted to in-sample estimates with the target data being the very same data used in the base model fitting... – jbowman Jun 12 '18 at 3:46
• So yes, it's quite likely that it will cause overfitting, unless the base models are rather incomplete and fill in each other's gaps nicely, e.g., $y = x_1 + e$, $y = x_2 + \epsilon$ as two base models would be OK. – jbowman Jun 12 '18 at 3:48
• Right. In fact, I tend to split into more than 2 groups, liking 10 groups for some reason. The base models are then trained with much more data so are more accurate (at least in situations where you don't have overwhelming amounts of data) and the blender is trained on predictions from models that have accuracy characteristics that are closer to what it will see when going operational. – jbowman Jun 12 '18 at 14:54
• Yes, it looks like you've got it, except that I accumulate all the 10 out of sample predictions and use them collectively, all at once, for training the blender. This is "stacking" rather than "blending"; see mlwave.com/kaggle-ensembling-guide, about 40% of the way down the page. If you are blending, the 2-split is probably better, because the blender has more data to train on. – jbowman Jun 12 '18 at 16:00
• Well stacking doesn't train the "blender" (now "stacker") ten times, it trains it once on the collected predictions from the 10 folds. But the base models are trained ten times for stacking and use to predict the 10 10% holdouts, so there's more work in that respect. – jbowman Jun 12 '18 at 16:29

Let me explain.

In stacking you split your data in two. One holdout set (10 - 20%). One training set (80 - 90%).

You train your base learners individually on the training set using the same cross-validation method for each. You must use the same cross-validation fold indices for all base learners. This is because you can only train the metaclassifier on the predicted probabilities of those test-fold sections and the original raw data associated these rows; because these rows weren't used for building the base-learners.

Note: Not a single training fold can overlap a testing fold. If row 37 is inside a test fold; it cannot occur in any training fold. Otherwise you will be passing information from layer to the next. Use 10 to 20% of the training set rows as test-folds.

Now use each trained base learning to predict on the holdup set, and write new probability variables to it (Example: SVM_probs_up, LDA_probs_up).

Once the metaclassifier is trained using the test-fold probabilities from each base learner + the raw data from the same rows, then use the meta-classier to predict on the holdout set for your final results.

With Blending the training set is also split. But instead of using cross-validation you just take out 30% - 40% straight away (training-validation-set) and then train the base-learners on the remaining 60% - 70% of the training set. Then predict to training-validation-set and holdout-set, writing these as new variables to both sets. Then train the metaclassifier on the training validation set, then predict using the metaclassifier on the holdout set for your final results.