Stacking without splitting data

Question

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for training of combiner of predictions (layer-2, right figure).

In my project, I have two different multi-classification models. And I have a dataset (train/dev/test) which was used for training and testing two models. When I have learned Stacking, I thought I tried to use the whole training set for the blending training set (layer-2), then, test the blender with the test data. Though I read the book and other websites, and they mention the training set is splitted into subsets.

Is it uncommon (or not recommended) to use the whole training set for both layer-1 and 2? I thought this is not wrong since test data has been already prepared. I have already trained my model by the whole training dataset. So if it is not recommended, should I train my models with splitted training dataset?

The images are cited from "Hands on Machine Learning with scikit-learn and Tensorflow." (2017).

Answer 1

You have to have real, out-of-sample, predictions as the input to your blender, otherwise your blender is not learning about, and thereby improving, prediction accuracy - but instead learning about, and thereby improving, in-sample estimation accuracy, which can lead to overfitting. This is why you cannot use the whole training set for both layers - if you do, some of the "predictions" made by the base models will actually be in-sample estimates, not out-of-sample predictions.

You split your training data set so that subset 1 is used to train your base models. This is what is shown in your left picture. Your base models then are used to generate predictions for subset 2, and these predictions, along with the actuals for subset 2, are given to your blender for training. This is what is shown in your right picture. Basically, the predictions are features that are given to your blender, along possibly with other features from subset 2.

The model that the blender comes up with based on subset 2 is then used to predict the test data. This can be done by predicting the test data with each of the base models (developed on subset 1), then predicting the test data again with the blender model (developed on subset 2) + the predictions from the base models. The resultant predictions are the ones you use for calibrating / testing the combined base models + blender model.

Alternatively, you can re-train your base models on subset 1 and 2 prior to making predictions for the test data set. This will tend to improve the base model predictions of the test data set, but (hopefully slightly) weaken the link between the base models and the blender model, as the blender saw less-accurate predictions when it was being trained. The blender will consequently add less value and more overfitting, but given that the base models are more accurate, it may balance out.

ETA (from comments): In practice, I tend to split into more than 2 groups, liking 10 groups for some reason. The base models are then trained with much more data so are more accurate (at least in situations where you don't have overwhelming amounts of data) and the blender is trained on predictions from models that have accuracy characteristics that are closer to what it will see when going operational, which is a win-win in accuracy terms.

Answer 2

Let me explain.

In stacking you split your data in two. One holdout set (10 - 20%). One training set (80 - 90%).

You train your base learners individually on the training set using the same cross-validation method for each. You must use the same cross-validation fold indices for all base learners. This is because you can only train the metaclassifier on the predicted probabilities of those test-fold sections and the original raw data associated these rows; because these rows weren't used for building the base-learners.

Note: Not a single training fold can overlap a testing fold. If row 37 is inside a test fold; it cannot occur in any training fold. Otherwise you will be passing information from layer to the next. Use 10 to 20% of the training set rows as test-folds.

Now use each trained base learning to predict on the holdup set, and write new probability variables to it (Example: SVM_probs_up, LDA_probs_up).

Once the metaclassifier is trained using the test-fold probabilities from each base learner + the raw data from the same rows, then use the meta-classier to predict on the holdout set for your final results.

With Blending the training set is also split. But instead of using cross-validation you just take out 30% - 40% straight away (training-validation-set) and then train the base-learners on the remaining 60% - 70% of the training set. Then predict to training-validation-set and holdout-set, writing these as new variables to both sets. Then train the metaclassifier on the training validation set, then predict using the metaclassifier on the holdout set for your final results.