Addressing model uncertainty

Question

I was wondering how the Bayesians in the CrossValidated community view the problem of model uncertainty and how they prefer to deal with it? I will try to pose my question in two parts:

1. How important (in your experience / opinion) is dealing with model uncertainty? I haven't found any papers dealing with this issue in the machine learning community, so I'm just wondering why.

2. What are the common approaches for handling model uncertainty (bonus points if you provide references)? I've heard of Bayesian model averaging, though I am not familiar with the specific techniques / limitations of this approach. What are some others and why do you prefer one over another?

Answer 1

There are two cases which arise in dealing with model-selection:

• When the true model belongs in the model space.

  This is very simple to deal with using BIC. There are results which show that BIC will select the true model with high probability.

However, in practice it is very rare that we know the true model. I must remark BIC tends to be misused because of this (probable reason is its similar looks as AIC). These issues have been addressed on this forum before in various forms. A good discussion is here.

• When the true model is not in the model space.

  This is an active area of research in the Bayesian community. However, it is confirmed that people know that using BIC as a model selection criteria in this case is dangerous. Recent literature in high dimension data analysis shows this. One such example is this. Bayes factor definitely performs surprisingly well in high dimensions. Several modifications of BIC have been proposed, such as mBIC, but there is no consensus. Green's RJMCMC is another popular way of doing Bayesian model selection, but it has its own short-comings. You can follow-up more on this.

There is another camp in Bayesian world which recommends model averaging. Notable being, Dr. Raftery.

• Bayesian model averaging.

  This website of Chris Volinksy is a comprehensive source of Bayesian model averging. Some other works are here.

Again, Bayesian model-selection is still an active area of research and you may get very different answers depending on who you ask.

Answer 2

A "true" Bayesian would deal with model uncertainty by marginalising (integrating) over all plausble models. So for example in a linear ridge regression problem you would marginalise over the regression parameters (which would have a Gaussian posterior, so it could be done analytically), but then marginalise over the hyper-paremeters (noise level and regularisation parameter) via e.g. MCMC methods.

A "lesser" Bayesian solution would be to marginalise over the model parameters, but to optimise the hyper-parameters by maximising the marginal likelihood (also known as the "Bayesian evidence") for the model. However, this can lead to more over-fitting than might be expected (see e.g. Cawley and Talbot). See the work of David MacKay for information on evidence maximisation in machine learning. For comparison, see the work of Radford Neal on the "integrate everything out" approach to similar problems. Note that the evidence framework is very handy for situations where integrating out is too computationally expensive, so there is scope for both approaches.

Effectively Bayesians integrate rather than optimise. Ideally, we would state our prior belief regarding the characteristics of the solution (e.g. smoothness) and make predictions notoionally without actually making a model. The Gaussian process "models" used in machine learning are an example of this idea, where the covariance function encodes our prior belief regarding the solution. See the excellent book by Rasmussen and Williams.

For practical Bayesians, there is always cross-validation, it is hard to beat for most things!

Answer 3

One of the interesting things I find in the "Model Uncertainty" world is this notion of a "true model". This implicitly means that our "model propositions" are of the form:

$$M_i^{(1)}:\text{The ith model is the true model}$$

From which we calculate the posterior probabilities $P(M_i^{(1)}|DI)$. This procedure seems highly dubious at a conceptual level to me. It is a big call (or an impossible calculation) to suppose that the $M_i^{(1)}$ propositions are exhaustive. For any set of models you can produce, there is sure to be an alternative model you haven't thought of yet. And so goes the infinite regress...

Exhaustiveness is crucial here, because this ensures the probabilities add to 1, which means we can marginalise out the model.

But this is all at the conceptual level - model averaging has good performance. So this means there must be a better concept.

Personally, I view models as tools, like a hammer or a drill. Models are mental constructs used for making predictions about or describing things we can observe. It sounds very odd to speak of a "true hammer", and equally bizzare to speak of a "true mental construct". Based on this, the notion of a "true model" seems weird to me. It seems much more natural to think of "good" models and "bad" models, rather than "right" models and "wrong" models.

Taking this viewpoint, we could equally well be uncertain as to the "best" model to use, from a selection of models. So suppose we instead reason about the propostion:

$$M_i^{(2)}:\text{Out of all the models that have been specified,}$$ $$\text{the ith model is best model to use}$$

Now this is a much better way to think about "model uncertainty" I think. We are uncertain about which model to use, rather than which model is "right". This also makes the model averaging seem like a better thing to do (to me anyways). And as far as I can tell, the posterior for $M_{i}^{(2)}$ using BIC is perfectly fine as a rough, easy approximation. And further, the propositions $M_{i}^{(2)}$ are exhaustive in addition to being exclusive.

In this approach however, you do need some sort of goodness of fit measure, in order to gauge how good your "best" model is. This can be done in two ways, by testing against "sure thing" models, which amounts to the usual GoF statistics (KL divergence, Chi-square, etc.). Another way to gauge this is to include an extremely flexible model in your class of models - perhaps a normal mixture model with hundreds of components, or a Dirichlet process mixture. If this model comes out as the best, then it is likely that your other models are inadequate.

This paper has a good theoretical discussion, and goes through, step by step, an example of how you actually do model selection.

Answer 4

I know people use DIC and Bayes factor, as suncoolsu said. And I was interested when he said "There are results which show that BIC will select the true model with high probability" (references?). But I use the only thing I know, which is posterior predictive check, championed by Andrew Gelman. If you google Andrew Gelman and posterior predictive checks you will find a lot of things. And I'd take a look at what Christian Robert is writting on ABC about model choice. In any case, here are some references I like, and some recent posts in Gelman's blog:

Blog

DIC and AIC; More on DIC. Model checking and external validation

Papers on posterior predictive checks:

GELMAN, Andrew. (2003a). “A Bayesian Formulation of Exploratory Data Analysis and Goodness-of-fit Testing”. International Statistical Review, vol. 71, n.2, pp. 389-382.

GELMAN, Andrew. (2003b). “Exploratory Data Analysis for Complex Models”. Journal of Computational and Graphic Statistics, vol. 13, n. 4, pp. 755/779.

GELMAN, Andrew; MECHELEN, Iven Van; VERBEKE, Geert; HEITJAN, Daniel F.; MEULDERS, Michel. (2005). “Multiple Imputation for Model Checking: Completed-Data Plots with Missing and Latent Data.” Biometrics 61, 74–85, March

GELMAN, Andrew; MENG, Xiao-Li; STERN, Hal. (1996). “Posterior Predictive Assessment of Model Fitness via Realized Discrepancies”. Statistica Sinica, 6, pp. 733-807.