I was hoping some of the more experienced ensemblers could help me with a couple of questions I have regarding stacking.

The assumption is that we have a classic train/test split, the goal is maximal test performance, and that we don't know whether stacking will be beneficial right from the start (so want to avoid partitioning train further, thus the CV-based stacking approach).

Here is my understanding steps used in stacking:

  1. Fix the fold indices at the start for train, e.g. using sklearn.model_selection.KFold in Python or caret::createFolds in R (e.g. we are using 5-fold CV - call this split fixed_5_folds)

  2. Build some tuned (first level) models using cross-validation (tuned using fixed_5_folds): M1, M2, M3

  3. Take these tuned models and generate your out-of-fold predictions (meta-features) for the training dataset (using fixed_5_folds?):

    meta_features_train = [META_M1_train, META_M2_train, META_M3_train]

  4. Re-fit models M1, M2 and M3 on the full train dataset (excluding the meta-features), predict for the test data to generate the equivalent meta-features for the test data:

    meta_features_test = [META_M1_test, META_M2_test, META_M3_test]

  5. Tune a meta-model (second level) on meta_features_train, using cross-validation to tune its hyper-parameters (using using fixed_5_folds?)

  6. Re-fit this tuned meta-model to the full meta_features_train dataset, predict for meta_features_test

  7. These predictions on meta_features_test are your final predictions

I have two main questions here:

  1. Is my understanding of the stacking process correct?

  2. Is it necessary/good/bad to use the exact same folds split throughout the entire analysis? E.g:

    a) Using different CV folds to generate the OOF predictions/meta-features in step 3, e.g. let's say I use a different split of 20 folds instead to generate my training OOF meta-features (Possible advantage: More data will be used to generate your meta-features)

    b) Using different CV folds to tune the meta-model in step 5, e.g. let's say I settle on Ridge Regression as the meta-model and the small number of meta-features means training is very fast, so I decide to instead use 3 repeats of 10-fold CV to tune this meta-model (Possible advantage: same reason repeated CV is used generally - reducing the variability of the meta-model CV estimates, allowing better hyper-parameter selection)

    c) Possible disadvantage: Varying the folds used could be introducing some sort of leakage that I'm not aware of

For point b) above, it seems like varying the folds seems to have been used by some well-known kaggle competitors, e.g. Giba in his 1st place solution here, although he goes from using 5 folds at the first level to 4 folds at the second level (the point being that his folds don't appear to be fixed between levels 1 and 2).

Hopefully you guys can help!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.