I'm trying to figure out the differences between the "Super Learner" approach of Van der Laan et al. (2007), the "Stacked regressions" approach of Leo Breiman (1996) and the "Stacked generalization" by Wolpert (1992).
In particular, I think they all describe as an option the following approach:
- do cross-validation (CV) with same fold splits for all models we want to consider,
- fit all our candidate models on the training part of each CV-split (possibly picking good hyperparameters by CV performance),
- obtain the out-of-fold predictions for the validation part of each CV-split for all models, and
- train a new model (e.g. weighted average or something else, aka the "second level model") that use the out-of-fold predictions as the features to predict the outcome of interest.
Am I right that all papers cover that? As far as I can see, Breiman acknowledges the origin of the idea with Wolpert, but then provides additional investigation/results and recommends a particular type of second level model (i.e. weighted averages). Is there an obvious reason why the "Super Learner" paper is not citing the other two papers, or did they likely just overlook it?
