Cross Validation in StackingClassifier Scikit-Learn

Question

In Scikit-Learn StackingClassifier documentation it's written:

Note that estimators_ are fitted on the full X while final_estimator_ is trained using cross-validated predictions of the base estimators using cross_val_predict.

... the default 5-fold cross validation

My question, why use 5-fold cross-validation only in the final estimator? why isn't final estimator fitted on the full X' (output from base estimators)?

Answer 1

This includes 2 questions, I will address each of them.

1. We could use cross-validation on the entire system, but that would handicap us a bit too much.

The purpose of cross-validation is to find the optimal parameters, those that allow the model to fit the data well without over-fitting. It suffices that our final estimator does this; there is no need for individually figuring out the settings of all the base estimators. The base estimators can include a bunch of different parameter settings, for example; as well as a selection of different types of classifiers. If any of them are prone to overfitting this should be offset by others not having that problem. As long as the final estimator does not put all of its eggs in the wrong basket, we should be fine (and this is why we need cross-validation here, to make sure this does not happen).

2. We will train the final estimator on the full training set -- this happens after we find the optimal parameters or set of base estimators using cross-validation. As the name says, cross-validation is meant for validating the method. Not for creating the final model.