I am learning to do stacking. I want to know what will the input be to level 2 classifiers In classification/regression problems, say if we use five different base classifiers, we get 5 predictions for each example. What would be the input to the second level classifiers?
 A: Say you have 5 base-estimators and 1 meta-estimator (that will be trained on the predictions of your 5 base-estimators).
The first step is to train your base estimators. You should think of it like a bit like performing a 5-fold cross validation. Split your training set into 5 parts and train each of the models on four of those parts, while holding the fifth out for prediction. Each model should be trained on a different combination of training folds and have a different fold for prediction. In detail:


*

*The 1st model: Training folds $(1, 2, 3, 4)$. Prediction fold: $5$

*The 2nd model: Training folds $(1, 2, 3, 5)$. Prediction fold: $4$

*The 3rd model: Training folds $(1, 2, 4, 5)$. Prediction fold: $3$

*The 4th model: Training folds $(1, 3, 4, 5)$. Prediction fold: $2$

*The 5th model: Training folds $(2, 3, 4, 5)$. Prediction fold: $1$


Now, by combining the predictions of the 5 base-estimators, we have the predictions of the whole dataset. These predictions will be the training set of the meta-estimator! Because these originated from the initial training set, we have their actual labels.
Finally, if we want to evaluate the performance of the stacked classifier, we need to follow the same path we did during training. Namely, the test data should be first passed to each of the base-estimators for prediction. Then, these predictions are averaged and passed as input to the meta-estimator. Finally, the predictions of the meta-estimator are compared with the actual labels for evaluation.
The whole procedure is depicted in the following figure:

You can also read this blog post for more information on model stacking.
A: Be careful with the information leakage of the method depicted above. It can give you much lower validation error vs test error and not generalize well. I experienced it and it was quite shocking, on a fairly small dataset.
See this post: https://www.kaggle.com/general/18793
