Does it make sense to do CV-error-weighted model averaging?

Question

We often average models together to create an aggregate prediction model. Some recent research suggests that simple model averages perform as well or better than model averages weighted by functions of information criterion scores. Weighted or simple, model averaging often (but not always) performs better than choosing the model with the best score. Many information criterion scores can be seen as approximations to different cross-validation schemes. Even though one model performs worse in cross-validation than another, it is still informative. I argue that an average model with weights as functions of cross-validation scores makes sense. Yet most people who do cross-validation to tune hyperparameters just choose the value of the hyperparameter that minimizes cross-validation error. In what circumstances might we expect a cross-validation-error-weighted average model to perform better than a single "best" model? I would love it if someone could point me to the most relevant research on this. Thank you.

Answer 1

There already exists a method very similar to what you're describing (weighted averages of predictions using cross-validated scores as weights). It is called SuperLearner.

http://biostats.bepress.com/ucbbiostat/paper222/

(This book focuses on another method called TMLE, but succinctly describes SuperLearner and its theory in a few chapters)

Googling SuperLearner will get you the latest research on that.

Answer 2

I argue that an average model with weights as functions of cross-validation scores makes sense.

Yes, but...

... only if you can be sure that your cross-validation measurement of the performance is good enough. In practice, e.g. I deal with sample sizes that are far too small to reliably compare models. Related questions and answers come up regularly.

IMHO, the basic rules for data-driven model optimization applies here as well.

So my advise would be: before trying to use any such scheme, make sure your measurement of model performance is precise enough to allow meaningful distinction between the models (or cases, if you boost).

Yet most people who do cross-validation to tune hyperparameters just choose the value of the hyperparameter that minimizes cross-validation error.

I suspect that far more people do so than have sufficient data to apply such a scheme reliably. See e.g. @DikranMarsupial's paper for a start:

• G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (pdf)

In what circumstances might we expect a cross-validation-error-weighted average model to perform better than a single "best" model?

I think this question is ill-posed as it is: by definition I'd say that you cannot top the best model. It doesn't matter how that model is derived.
I guess a better question may be in which situations we have to expect that

• on the one hand, we cannot rely on getting a sufficiently good single model, but
• on the other hand cross-validation is reliable enough to allow us to derive good weights for the averaging.

I'd guess that this margin is rather narrow (if it actually exists in practice): Averaging (weighted or not) can only help to reduce errors caused by model instability. Good performance estimates need sufficient sample size. So situations with high-dimensional data and sample sizes large enough to allow good performance estimation while not yet reaching the sample size sufficient to produce stable models would be the candidate. I'd go for situations that actually require complex modes, because otherwise reducing model complexity/typical regularization would be used.
I'd also guess that the weighting needs a whole lot of care to avoid overfitting (compare to the problems of boosting).