Suppose there exist n individual classifiers which have different parameters and have been trained on the same data. In order to build an ensemble of these classifiers, is it an optimal method to assign weights to these individual classifiers based on their cross validation accuracy on the training data itself? Are the weights assigned in such a way going to be optimal when the ensemble is tested on the unknown test data?

Thank you.


Boosting is a strategy that uses such a heuristic. There's quite a body of literature about how and when it helps.

As for whether it is optimal for generalization on unknown test data: boosting is known to be more likely to overfit than e.g. an aggregation scheme without weighting. However, there's no free lunch, and no universal optimal heuristic here.

In particular, if you encounter a model with bad performance, there are totally opposite but equally sensible steps you could take:

  • you already suggested that you should downweight models with low performance
  • but low performance can also be an indicator of a reasonably well working algorithm that encountered difficult training and test data. And in that case you may want to upweight it.

Keep in mind that unless you have large sample size (which is typically not the situation where much thought is spent on ensemble models), the variance uncertainty due to finite test sample is often quite large. If this is non-negligible, boosting may lead to overfitting. The boosted ensemble may overestimate class separation.

Here's some literature:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.