Recently, I've become interested in model stacking as a form of ensemble learning. In particular, I've experimented a bit with some toy datasets for regression problems. I've basically implemented individual "level 0" regressors, stored each regressor's output predictions as a new feature for a "meta-regressor" to take as its input, and fit this meta-regressor on these new features (the predictions from the level 0 regressors). I was extremely surprised to see even modest improvements over the individual regressors when testing the meta-regressor against a validation set.
So, here's my question: why is model stacking effective? Intuitively, I would expect the model doing the stacking to perform poorly since it appears to have an impoverished feature representation compared to each of the level 0 models. That is, if I train 3 level 0 regressors on a dataset with 20 features, and use these level 0 regressors' predictions as input to my meta-regressor, this means my meta-regressor has only 3 features to learn from. It just seems like there is more information encoded in the 20 original features that the level 0 regressors have for training than the 3 output features that the meta-regressor uses for training.