After reading a bit on ensemble learning methods in my spare time, I took a quick glance at the Wikipedia page and came across this explanation on a supposedly common technique known as a "bucket of models". The explanation, given here, is as follows:

A "bucket of models" is an ensemble technique in which a model selection algorithm is used to choose the best model for each problem. When tested with only one problem, a bucket of models can produce no better results than the best model in the set, but when evaluated across many problems, it will typically produce much better results, on average, than any model in the set.

The most common approach used for model-selection is cross-validation selection (sometimes called a "bake-off contest"). It is described with the following pseudo-code:

For each model m in the bucket:
  Do c times: (where 'c' is some constant)
    Randomly divide the training dataset into two datasets: A, and B.
    Train m with A
    Test m with B
Select the model that obtains the highest average score

Cross-Validation Selection can be summed up as: "try them all with the training set, and pick the one that works best".

My question is how this would actually be implemented in practice. Say I have a typical regression problem where I want to predict the values of some response, and I train an arbitrary amount of different models using a variety of different algorithms that take the same inputs as in my training set (the bucket). Assume that I have perhaps 15 combinations of the same predictors in my training set in my prediction set. Is this passage implying that each model in the bucket will be used to compute a single prediction for each combination (so one of the fifteen in my prediction set) and somehow, there is a method to figure out what is the "best" model for each combination (essentially, each "problem" is a single combination of the fifteen)? That is, each prediction required out of the fifteen is potentially being predicted by a completely different model versus one model predicting all values? Or, is the correct interpretation as follows; that the problem is the complete regression on the response that I desire, and the bucket of models is a different way of saying "just train a bunch of different models and choose the best one overall for the regression on all combinations"?

I feel like the latter definition must be the correct interpretation, because how are we supposed to figure out the "best" model for each combination in the prediction set when we don't know the true labels? We can easily do this for the second interpretation. The definition further talks about gating with a perceptron, which would then make the first interpretation make a bit more sense to me.

Thanks for any responses.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.