Model Stacking Train Test Split Methdods

Question

I am trying to validate my processes in terms of how I am engaging in model stacking for binary classification. Say I have two models as my base models, models A and B both with different classifiers and model C as my meta model.

My steps are as below.

1. Split data into train, test sets
2. Split my test set into a validation set (for my meta model training), and a final testing set.
3. Train my base models A and B, on the training set using CV. Return the test statistics on the base models on my final testing set.
4. Train my meta model using the probabilities generated in the base models on the validation set in part 2. Return the test statistics on the meta model on the final testing set.
5. Compare the testing results from models A,B,C.

Answer 1

These steps appear reasonably coherent for setting up a model stacking workflow for binary classification. Two minor points and a medium one:

1. It is unclear if feature engineering is done for the metamodel C, that could be beneficial.
2. The metamodel C could also be trained using cross-validation.
3. The base models A and B are trained using CV on the training set (which is correct), but when generating predictions for metamodel C, we only need to ensure these predictions are out-of-fold predictions. I think in the quest to avoid data leakage, we are using an unnecessarily small training set for the metamodel.

The two most straightforward reference on stacking I have seen are:

1. Machine Learning Design Patterns by Lakshmanan et al. (Nice but not too in-depth)
2. The Kaggle Book by Banachewicz & Massaron. (Quite nice - see for example the Diagram of a two-layer stacking process with final averaging of predictions)

Both books cover considerations around stacking in reasonable detail and even provide code examples. Use those before getting your own variants. The sklearn documentation example on: Combine predictors using stacking is very good too.