I'm trying to figure out the differences between the "Super Learner" approach of Van der Laan et al. (2007), the "Stacked regressions" approach of Leo Breiman (1996) and the "Stacked generalization" by Wolpert (1992).

In particular, I think they all describe as an option the following approach:

  • do cross-validation (CV) with same fold splits for all models we want to consider,
  • fit all our candidate models on the training part of each CV-split (possibly picking good hyperparameters by CV performance),
  • obtain the out-of-fold predictions for the validation part of each CV-split for all models, and
  • train a new model (e.g. weighted average or something else, aka the "second level model") that use the out-of-fold predictions as the features to predict the outcome of interest.

Am I right that all papers cover that? As far as I can see, Breiman acknowledges the origin of the idea with Wolpert, but then provides additional investigation/results and recommends a particular type of second level model (i.e. weighted averages). Is there an obvious reason why the "Super Learner" paper is not citing the other two papers, or did they likely just overlook it?

  • 1
    $\begingroup$ This sounds a bit gossipy... But I've been reading CV threads on some of these topics and came across this. See the claim "(...) that van der Laan has a tendency to invent new names for already existing ideas (e.g. the super-learner)... " $\endgroup$
    – dipetkov
    Jan 27 at 20:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.