I recently read this blog and it has many ideas for ensembling various models. I created three models for my training data, random forest model, SVM model and a KNN model. However when I use linear method to stack these ensemble methods together the error I get is lower than the RF model only about 40% of the time.
The linear method that I employ for stacking is a plain one: determining coefficients of each model and minimizing their least squares.
Am I missing something? This results in a "best case" scenario less than half the time, so I would be better off just using the randomforest model.
Secondly, What exactly are meta features as described in this paper? How do I extract them for any data and successfully use them to build a lower error ensemble that individual models themselves?