I was hoping some of the more experienced ensemblers could help me with a couple of questions I have regarding stacking.
The assumption is that we have a classic
test split, the goal is maximal
test performance, and that we don't know whether stacking will be beneficial right from the start (so want to avoid partitioning
train further, thus the CV-based stacking approach).
Here is my understanding steps used in stacking:
Fix the fold indices at the start for
train, e.g. using
R(e.g. we are using 5-fold CV - call this split
Build some tuned (first level) models using cross-validation (tuned using
Take these tuned models and generate your out-of-fold predictions (meta-features) for the training dataset (using
meta_features_train = [META_M1_train, META_M2_train, META_M3_train]
M3on the full
traindataset (excluding the meta-features), predict for the
testdata to generate the equivalent meta-features for the
meta_features_test = [META_M1_test, META_M2_test, META_M3_test]
Tune a meta-model (second level) on
meta_features_train, using cross-validation to tune its hyper-parameters (using using
Re-fit this tuned meta-model to the full
meta_features_traindataset, predict for
These predictions on
meta_features_testare your final predictions
I have two main questions here:
Is my understanding of the stacking process correct?
Is it necessary/good/bad to use the exact same folds split throughout the entire analysis? E.g:
a) Using different CV folds to generate the OOF predictions/meta-features in step 3, e.g. let's say I use a different split of 20 folds instead to generate my training OOF meta-features (Possible advantage: More data will be used to generate your meta-features)
b) Using different CV folds to tune the meta-model in step 5, e.g. let's say I settle on Ridge Regression as the meta-model and the small number of meta-features means training is very fast, so I decide to instead use 3 repeats of 10-fold CV to tune this meta-model (Possible advantage: same reason repeated CV is used generally - reducing the variability of the meta-model CV estimates, allowing better hyper-parameter selection)
c) Possible disadvantage: Varying the folds used could be introducing some sort of leakage that I'm not aware of
For point b) above, it seems like varying the folds seems to have been used by some well-known kaggle competitors, e.g. Giba in his 1st place solution here, although he goes from using 5 folds at the first level to 4 folds at the second level (the point being that his folds don't appear to be fixed between levels 1 and 2).
Hopefully you guys can help!