Model stacking, Super Learner Algorithm

Question

I've recently started studying ensembles in ML, particularly Super Learner Algorithm. To be honest, although I have read several articles related to this topic, I am a little bit confused. I want to go step by step and do everything manually, so that I can truly understand the process.

The algorithm is usually described as the sequence of the following steps:

1. Train each of the L base algorithms on the training set.
2. Perform k-fold cross-validation on each of these learners and collect the cross-validated predicted values from each of the L algorithms.
3. The N cross-validated predicted values from each of the L algorithms can be combined to form a new N x L matrix. This matrix, along wtih the original response vector, is called the "level-one" data. (N = number of rows in the training set)
4. Train the metalearning algorithm on the level-one data.
5. The "ensemble model" consists of the L base learning models and the metalearning model, which can then be used to generate predictions on a test set.

I have several questions:

1. Why the first and the second steps are separated? For simplicity lets assume that I needn't tune any parametres. Does it mean that I just have to train a model, xgbTree, for example, using k-fold CV? E.g:

tc_XGB <- trainControl(method = "cv", number = 5, savePred = TRUE)

fit_XGB <- train(x = input_x, y = input_y, method = "xgbTree",
           trControl = train_control_final, tuneGrid = Grid_final)

Note: input_x and input_y are from a training set.

2. The next step is to collect the cross-validated predicted values. Should I use fit_XGB $pred and extract all cross-validated predictions and repeat this action L times (L - a number of algorithms)?

I would say that all the next steps are more or less clear to me. I have got some doubts, however, I really can't put my finger on what is wrong with my approach.

Answer 1

Question 1

I think the confusion arises from common practice making the separation unclear: "train a model...using k-fold CV" isn't actually a thing. A $k$-fold cross-validation doesn't produce a trained model. However, in most hyperparameter tuning packages, caret included, a final model is automatically trained on the entire training set after the cross-validation picks a best hyperparameter set; from the docs:

The combination with the optimal resampling statistic is chosen as the final model and the entire training set is used to fit a final model.

So, caret is doing both steps 1 and 2 for you.

Question 2

Yes, that will work. See e.g. How does cross-validation in train (caret) precisely work?.

You said you wanted to do this manually to understand, which is good; but I will mention that there is a package extending caret to perform all this automatically: caretEnsemble.