I've recently started studying ensembles in ML, particularly Super Learner Algorithm. To be honest, although I have read several articles related to this topic, I am a little bit confused. I want to go step by step and do everything manually, so that I can truly understand the process.

The algorithm is usually described as the sequence of the following steps:

  1. Train each of the L base algorithms on the training set.
  2. Perform k-fold cross-validation on each of these learners and collect the cross-validated predicted values from each of the L algorithms.
  3. The N cross-validated predicted values from each of the L algorithms can be combined to form a new N x L matrix. This matrix, along wtih the original response vector, is called the "level-one" data. (N = number of rows in the training set)
  4. Train the metalearning algorithm on the level-one data.
  5. The "ensemble model" consists of the L base learning models and the metalearning model, which can then be used to generate predictions on a test set.

I have several questions:

  1. Why the first and the second steps are separated? For simplicity lets assume that I needn't tune any parametres. Does it mean that I just have to train a model, xgbTree, for example, using k-fold CV? E.g:
tc_XGB <- trainControl(method = "cv", number = 5, savePred = TRUE)

fit_XGB <- train(x = input_x, y = input_y, method = "xgbTree",
           trControl = train_control_final, tuneGrid = Grid_final)

Note: input_x and input_y are from a training set.

  1. The next step is to collect the cross-validated predicted values. Should I use fit_XGB $pred and extract all cross-validated predictions and repeat this action L times (L - a number of algorithms)?

I would say that all the next steps are more or less clear to me. I have got some doubts, however, I really can't put my finger on what is wrong with my approach.


1 Answer 1


Question 1

I think the confusion arises from common practice making the separation unclear: "train a model...using k-fold CV" isn't actually a thing. A $k$-fold cross-validation doesn't produce a trained model. However, in most hyperparameter tuning packages, caret included, a final model is automatically trained on the entire training set after the cross-validation picks a best hyperparameter set; from the docs:

The combination with the optimal resampling statistic is chosen as the final model and the entire training set is used to fit a final model.

So, caret is doing both steps 1 and 2 for you.

Question 2

Yes, that will work. See e.g. How does cross-validation in train (caret) precisely work?.

You said you wanted to do this manually to understand, which is good; but I will mention that there is a package extending caret to perform all this automatically: caretEnsemble.

  • $\begingroup$ Ben Reiniger, thanks a lot for your answer. I've recently played around with some caret functions and found that using set.seed(...) produces similar CV folds, so that I can stack my CV predictions straight away. I completely forgot that caret trains the final model for me, which is truly useful. Thanks one more time! $\endgroup$
    – User
    Aug 31, 2020 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.