10
$\begingroup$

I'm wondering in what applications or scenarios (or in trying to answer what kind of questions), the researcher would prefer using Model Selection (such as AIC or BIC) over Model Averaging (such as Bayesian model averaging)? One example: if my question at hand is what's the most appropriate model, then I must use Model Selection. Another (not sure) example: I guess if we want less bias, Model Selection is also preferred. It would be great if you have other examples like these. Paper or book recommendations are also welcome.

$\endgroup$

4 Answers 4

14
$\begingroup$

Model specification is a better approach than either model selection (by which most people refer really to feature selection) or model averaging. Here are some pros and cons.

  • model specification requires much more up-front thinking with regard to nonlinearities, non-additivities (interactions), and model components to penalize but this thinking pays off in downstream computation and interpretation and results in accurate estimates of uncertainty. One example of model specification would be inclusion of a sensible number of nonlinear basis functions to allow for flexible modeling without assuming linearity, and specifying skeptical prior distributions for non-additive effects in a Bayesian model (or parameters to penalize in a frequentist setting, guided by cross-validation to get penalized maximum likelihood estimation)
  • model (feature) selection does not work as advertised as you will almost always be disappointed to find that the "found" features do not replicate in future samples, collinearities almost destroy the ability to select, and predictive performance is not as good as fitting full models with appropriate penalization (e.g., ridge regression typically predicts better than lasso or elastic net). Feature selection results in an example model not the model.
  • model averaging accomplishes the same thing as a carefully pre-specified very flexible single model, but with a lot more work and difficulty in interpretation. That is, if the domain of models being averaged are from the same family. Sometimes model averaging over different model families can take care of model family uncertainty though. For example you may be unsure of the link function to use when Y is binary and you are developing a probability model. Or in a time-to-event analysis you might ponder a proportional hazards model vs. an accelerated failure time model. But judicious use of extra parameters in any one of these models should also be considered. For example I can estimate risk ratios accurately from an odds ratio-based logistic regression model if I include approximately correct interactions in the logistic model. Examples of this may be found at fharrell.com.
$\endgroup$
1
  • $\begingroup$ Thank you so much, Frank! This is extremely helpful! $\endgroup$ Mar 9 at 3:45
11
$\begingroup$

Single models are much easier to interpret than ensembles. They are therefore often more easily accepted by non-technical users, and are easier to troubleshoot (e.g., when a prediction is way off).

Averaged models are much harder to apply null hypothesis significance testing to. This is a very hard criterion in many domains where publishability hinges on p values.

$\endgroup$
3
  • 2
    $\begingroup$ This is a really nice, concise answer. Do averaged models usually fit the data better (lower MSE or whatever metric you're using)? $\endgroup$
    – Eli
    Mar 3 at 14:09
  • 1
    $\begingroup$ @Eli: it's a common finding in forecasting that ensemble methods yield better forecasts. (Forecasters usually don't care about in-sample fit, which is not a good guide to out-of-sample forecasting performance.) $\endgroup$ Mar 3 at 14:42
  • $\begingroup$ @StephanKolassa Thanks a lot, Stephan! Your answer gives great guidance and is very insightful. I benefited a lot. $\endgroup$ Mar 9 at 3:50
3
$\begingroup$

Model averaging is more practical In applications where you don’t have resources to select a model and no need to explain the results in detail. Model selections is much more than comparing AICs, because if that’s all you do to select the models then don’t bother, just average.

A proper selection is a very time consuming expensive process. You can get a PhD working for months to come up with a parsimonious specification, that’s hundreds of thousands $$$, plus ongoing maintenance and opportunity cost of time to market off your product. There are situations when this can be afforded. An example is financial risk management where the cost of error can be existential to the firm, plus you have the regulators scrutinizing the models who also want you to explain the results to minute details. A counter example is marketing and sales, where instead of meticulous model selection you maybe better off with data mining and experimentation.

$\endgroup$
1
  • $\begingroup$ I see. Thanks a lot. It's very helpful! $\endgroup$ Mar 9 at 3:51
3
$\begingroup$

(This is a very rough thought still, please take it with a grain of salt)

Averaging is a viable strategy against random uncertainty, i.e. variance. OTOH, if the dominating problem of the situation is bias, a selection approach may work.

In practice, the situation may be even more blurred by the additional random uncertainty/variance that stems from judging the "health" of the model with only a limited number of independent cases (for the data I work with, number of idependent cases is often << number of rows in the data).

I'd therefore make the arguement the other way round compared to your example (and rather go in the specification direction Frank Harrell brought to your attention): selection (for low bias) has the prerequisite that variance uncertainty must be low.

$\endgroup$
2
  • $\begingroup$ +1. Your "very rough thought" is very insightful. $\endgroup$
    – usεr11852
    Mar 4 at 14:53
  • $\begingroup$ @cbeleitesunhappywithSX Thank you very much! This is very helpful! $\endgroup$ Mar 9 at 3:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.