Model Stacking - Gives poor performance I'm trying model stacking in a kaggle competition. However, what the competition is trying to do is irrelevant.
I think my approach of doing model stacking is not correct.
I have 4 different models:


*

*xgboost model with dense features (numbers, that can be ordered).

*adaboost model with sparse features (non-numeric features, which are label encoded, then one hot encoded).

*xghoost model with dense features (sentiment analysis using nltk's vader on text).
These models generate their probabilities of a multi-class problem, and feeds into a final neural network model that combine their results, and then generate another set of probabilities of a multi class problem.
However, the more models I tried to munge in creates a worse model. For example, If I only use the first model, I would get 73% accuracy, but with each model added, it would drop to less than 70% accuracy, with the score on kaggle increasing from 0.6X to 1.0 above.
Is this approach incorrect?
 A: It sounds like you may not be generating the "probabilities" (aka "level-one" data) correctly.  These predicted values should be cross-validated predicted values from the base learners (or sometimes people use a separate hold-out set to generate these predicted values).  My guess is that you are using predictions generated exclusively from the training set, which is leading to overfitting.
Here are some references which explain the construction of the level-one dataset in more detail:


*

*Scalable Ensemble Learning in H2O (Strata San Jose 2016) Slides

*useR! Machine Learning Tutorial: Stacking

*h2oEnsemble Stacking Tutorial

*"Scalable Ensemble Learning and Computationally Efficient Variance Estimation" (PhD Thesis, see chapter 2)
Soon, we will release H2O with XGBoost support, so you should be able to ensemble XGBoost models much more easily using the Stacked Ensemble method in H2O.  Or you could use H2O models for the time-being and skip the manual construction of the ensemble.
A: Stacking can give poor performance relative to the base models if a lot of overlap exists in the correct predictions of the ensembled models. Also, stacking tends to do better with a larger number of input models than with a smaller number of ensembled models.
A: It is quite easy to mess up the first stage model or to fail to see the leakage of informations when working with large blends of models. As stated by @Erin LeDell you should make sure that the second stage is learned from cross validated predictions of the first stage. I wrote the following tutorials regarding blending if you are interested:
Introduction to blending in Python (method and implementation oriented)
Why does blending works ? (theoretical arguments about the success of this method)
