4
$\begingroup$

In Scikit-Learn StackingClassifier documentation it's written:

Note that estimators_ are fitted on the full X while final_estimator_ is trained using cross-validated predictions of the base estimators using cross_val_predict.

... the default 5-fold cross validation

My question, why use 5-fold cross-validation only in the final estimator? why isn't final estimator fitted on the full X' (output from base estimators)?

$\endgroup$
1
$\begingroup$

This includes 2 questions, I will address each of them.

  1. We could use cross-validation on the entire system, but that would handicap us a bit too much.

The purpose of cross-validation is to find the optimal parameters, those that allow the model to fit the data well without over-fitting. It suffices that our final estimator does this; there is no need for individually figuring out the settings of all the base estimators. The base estimators can include a bunch of different parameter settings, for example; as well as a selection of different types of classifiers. If any of them are prone to overfitting this should be offset by others not having that problem. As long as the final estimator does not put all of its eggs in the wrong basket, we should be fine (and this is why we need cross-validation here, to make sure this does not happen).

  1. We will train the final estimator on the full training set -- this happens after we find the optimal parameters or set of base estimators using cross-validation. As the name says, cross-validation is meant for validating the method. Not for creating the final model.
$\endgroup$
1
  • $\begingroup$ I see.., overfitting in each base estimators is not really a problem because there are some estimators. So I think it's okay if we use cross-validation at the "outside" of the stacking model, right? I mean, both base and final estimators are trained with full $X_i$, where $X_i$ is the train set in the $i$-th fold $\endgroup$ – malioboro Aug 30 '20 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.