I'm trying model stacking in a kaggle competition. However, what the competition is trying to do is irrelevant.

I think my approach of doing model stacking is not correct.

I have 4 different models:

  1. xgboost model with dense features (numbers, that can be ordered).

  2. adaboost model with sparse features (non-numeric features, which are label encoded, then one hot encoded).

  3. xghoost model with dense features (sentiment analysis using nltk's vader on text).

These models generate their probabilities of a multi-class problem, and feeds into a final neural network model that combine their results, and then generate another set of probabilities of a multi class problem.

However, the more models I tried to munge in creates a worse model. For example, If I only use the first model, I would get 73% accuracy, but with each model added, it would drop to less than 70% accuracy, with the score on kaggle increasing from 0.6X to 1.0 above.

Is this approach incorrect?

  • $\begingroup$ does this accuracy based on the training or testing dataset? $\endgroup$ – itdxer Mar 29 '17 at 15:11
  • $\begingroup$ @itdxer Testing $\endgroup$ – user1157751 Mar 29 '17 at 15:11
  • $\begingroup$ so what about training, is it going up? maybe you just overfitting $\endgroup$ – itdxer Mar 29 '17 at 15:12
  • $\begingroup$ @itdxer Nope, the general direction is downtrend. Are multiple models ever used? $\endgroup$ – user1157751 Mar 29 '17 at 15:14

It sounds like you may not be generating the "probabilities" (aka "level-one" data) correctly. These predicted values should be cross-validated predicted values from the base learners (or sometimes people use a separate hold-out set to generate these predicted values). My guess is that you are using predictions generated exclusively from the training set, which is leading to overfitting.

Here are some references which explain the construction of the level-one dataset in more detail:

Soon, we will release H2O with XGBoost support, so you should be able to ensemble XGBoost models much more easily using the Stacked Ensemble method in H2O. Or you could use H2O models for the time-being and skip the manual construction of the ensemble.

| cite | improve this answer | |
  • $\begingroup$ For the first level, I split my data 90/10 for training and testing. I created my 1st layer model with 90% of available data with CV. Then I use the 1st layer models to create a final layer. Do you mean I should somehow combine the whole process into one step? $\endgroup$ – user1157751 Mar 29 '17 at 22:53
  • $\begingroup$ You should always combine the entire process into one step. Every decision you make should always be included in a validtion loop. $\endgroup$ – Matthew Drury Mar 30 '17 at 0:16
  • $\begingroup$ @user1157751 So you are saying that you're training your metalearner (the DNN in your case) on 90% of the data, which is the cross-validated predictions from the base models? If so, then your method is okay. I assume you are leaving the 10% test set for the very end to evaluate the performance of the ensemble. Perhaps your DNN metalearner is poorly trained and a GLM will work better as a metalearner. I find that unless you really spend some effort tuning the DNN, you will get better results with a GLM for the metalearning step. $\endgroup$ – Erin LeDell Mar 30 '17 at 0:21
  • $\begingroup$ @user1157751 What is the dimension of your level-one data? It should be n x L where n = number of training observations and L = number of base learners in the ensemble (in your case L = 3). $\endgroup$ – Erin LeDell Mar 30 '17 at 0:22
  • 1
    $\begingroup$ I'm training my DNN on 90% of the data, which is CV predictions ok my base models. I'm also leaving 10% test set for the very end. I tried another XGBoost at the end instead of DNN, and the performance was even worse. My n is around 50000. I'm thinking there might be something wrong with my feature extraction? I tried a XGBoost model with all features and it performed even worse than stacked model (both DNN/XGBoost). Does that give you any clue? $\endgroup$ – user1157751 Mar 30 '17 at 0:44

Stacking can give poor performance relative to the base models if a lot of overlap exists in the correct predictions of the ensembled models. Also, stacking tends to do better with a larger number of input models than with a smaller number of ensembled models.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.