2
$\begingroup$

Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"?

It would seem that model selection does indeed fit that definition. My thinking is that the model selection process considers a set of models and at every point in the sample space chooses the model from the model set which optimizes some criterion at that point. The result is thus a new model, defined piece-wise across the sample space according to the selection criterion.

Is this correct, or am I missing something here? I can't seem to find a definitive answer to this question in the literature, hence my post, but the paper "Estimation and Accuracy after Model Selection" by Bradley Efron seems to support this viewpoint. I've also heard of ensemble learning via a "bucket of models" (a description of which can be found on the "Ensemble Learning" wikipedia page) which seems related to what I am asking about, but I could just be misunderstanding that too.

Edit: Upon further reflection, it seems like frank's answer answers my question. By thinking of estimation as a specific kind of prediction problem (in the sense of using data to predict\guess what the parameter value is), it seems like model selection can be thought of as a specific kind of mixture of experts, which in turn is a specific kind of ensemble learning. Thus, model selection is a kind of ensemble learning.

Improve this question
$\endgroup$
1
  • $\begingroup$ Note that it is not uncommon for one member of the ensemble to be horribly overfitted, and to have its predictions receive too much weight in the ensemble such that the ensemble predictions are overfitted. $\endgroup$
    – Frank Harrell
    Jul 16, 2022 at 12:01

1 Answer 1

Reset to default
2
$\begingroup$

Your description of "ensemble" sounds like what is usually referred to as "mixture of experts", which is a special type of ensemble.

In a mixture of experts, one applies one of a collection of models depending on some properties of the input. But the notion of "ensembles of models" includes many more techniques. E.g., a simple standard ensemble in regression would be to take the average of the output of all the member models of an ensemble. More complex ensembles use boosting methods like e.g. the gradient boosting machines. For an overview of possible ensemble techniques, see here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I looked into these MoE models you mentioned, and it seems like they are concerned with taking a bunch of models and using them to define a new model piece-wise over the input space. This is a bit different from what I meant by "piece-wise" in my question, which is more so the idea of taking a family of estimators (using statistical language here) and using them to define a new estimator piece-wise over the sample space being considered (i.e. the space of possible observations you are considering), which hopefully has improved statistical properties. $\endgroup$
    – Euphoric Swole
    Jul 17, 2022 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.