I've recently started studying ensembles in ML, particularly Super Learner Algorithm. To be honest, although I have read several articles related to this topic, I am a little bit confused. I want to go step by step and do everything manually, so that I can truly understand the process.
The algorithm is usually described as the sequence of the following steps:
- Train each of the L base algorithms on the training set.
- Perform k-fold cross-validation on each of these learners and collect the cross-validated predicted values from each of the L algorithms.
- The N cross-validated predicted values from each of the L algorithms can be combined to form a new N x L matrix. This matrix, along wtih the original response vector, is called the "level-one" data. (N = number of rows in the training set)
- Train the metalearning algorithm on the level-one data.
- The "ensemble model" consists of the L base learning models and the metalearning model, which can then be used to generate predictions on a test set.
I have several questions:
- Why the first and the second steps are separated? For simplicity lets assume that I needn't tune any parametres. Does it mean that I just have to train a model, xgbTree, for example, using k-fold CV? E.g:
tc_XGB <- trainControl(method = "cv", number = 5, savePred = TRUE) fit_XGB <- train(x = input_x, y = input_y, method = "xgbTree", trControl = train_control_final, tuneGrid = Grid_final)
input_y are from a training set.
- The next step is to collect the cross-validated predicted values. Should I use
fit_XGB $predand extract all cross-validated predictions and repeat this action L times (L - a number of algorithms)?
I would say that all the next steps are more or less clear to me. I have got some doubts, however, I really can't put my finger on what is wrong with my approach.