Why in the stacking of scikit-learn the estimators are fitted on the whole training data?

Question

In chapter 7 of "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow", the first step of stacking method is spliting the train data into two subsets. The first subset is used to train the predictors in the first layer. Next, the first layer’s predictors are used to make predictions on the second (held-out) set and then create a new training set using these predicted values as input features. Use this new training set to train a blender：

But in Stacked generalization part of scikit-learn: "During training, the estimators are fitted on the whole training data X_train. They will be used when calling predict or predict_proba", it seems that scikit-learn use all data to train the "first layer" predictors. Why scikit-learn use this setting?

Answer 1

For an exact answer to the why question you would need to ask the scikit-learn developers about their motivation, we could only guess. In general, stacking means training several models on the data and having a meta-model that is trained on the predictions. Aurélien Géron mentions that it is a "common approach is to use a hold-out set", but it is not the only way. Another possible approach is to use cross-validation, in fact, this is what scikit-learn does.

Why did they decide to use cross-validation rather than two separate sets? Cross-validation re-uses the data, so it is better suitable for smaller datasets. For using two sets, you would need a large dataset so that each of the subsets would be big enough to train a model. It seems to be a safer default for general use software, while as noticed by Aurélien Géron, the other approach can be easily implemented if you want to use it.

Answer 2

As in Tim's answer, obtaining k-fold cross-validated predictions instead of the single split of data gives each estimator in the ensemble more data to learn from. It is important to not use predictions from the base learners on their own training set, lest the meta-estimator just copy the most-overfit base learner, but using k-fold takes care of that (see the next sentence from the paragraph you quoted).

When the docs say the base estimators are trained on the entire dataset, it means for the final ensemble ("when calling predict or predict_proba"). Again the point is that more data is better, and since the meta-estimator is already trained, the overfitting question is avoided. Besides, with the k-fold predictions of the base estimators, we otherwise don't have a single trained version of each base model anymore, so something needs to be retrained in any case.