CV (or Nested CV) are normally done to evaluate and compare different ML algorithms as part of model development and evaluation phases. Once these stages are complete, one normally develops the final model, training it on the entire dataset.

With that in mind, I am wondering how that affects final model creation for stacking since we have multiple layers of models. Every post I see (for example this user guide: http://rasbt.github.io/mlxtend/user_guide/classifier/StackingCVClassifier/ or this post: https://www.kaggle.com/general/18793) mention different reasons why stacking CV strategies are better than others but none go over the final stacked model selection.

So my question is, once stacking CV is complete, should one keep the best hyperparameters on the layer 2 model and retrain the stacking model on the entire dataset?

Should one discard the best hyperparameters on layer 1 models or would those still matter (I think they shouldn't matter due to the uncertainty similarly to how the best hyperparameters in Nested CV subfolds don't matter).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.