stacking and blending of regression models

Question

I am self-studying blending and stacking, and am especially interested in this in the context of regression models. I have been reading a number of the stacking, blending and bagging links posted on this forum, but have failed to find or (more likely) understand how to link the articles that mostly talk about classification to the field of regression. To give an example: Imagine one has 5 different regression predictions (e.g. 50,51,55,60,48), as well as the actual value (e.g. 53) for a range of datapoints. What is a simple way to stack these 5 predictions together? Can one simply use linear regression with appropriate cross-validation? I am somewhat unsure whether the fact that the regression coefficients or the implicit mixture weights do not add up to one is a problem in this regard, or whether can happily ignore this.

Answer 1

The simplest way to stack your predictions is to take the average. Linear regression is certainly an alternative. Here is a link to a video in which Phil Brierley describes using regularized regression instead of linear regression to combine model predictions. You could also look at accounts by other Kaggle winners. For example, the winner of the Bulldozer Price Prediction contest combined models "using a neural network".

I think the best thing to do is to look at things like Kaggle contest writeups and see what people did (or look at their code.)

As for your question, if you are only interested in prediction (which you probably are if you are stacking) then it doesn't matter if the mixture weights don't add up to 1.