Bayesian methods are about averaging over uncertainty rather than optimization. Explain?

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is?

My confusion is that in Bayesian model selection, if you compute the model evidence, surely that involves some element of optimization... you then compute the posterior probability to get the Bayes factor, for example, but it seems there is some element of optimization still. Is this incorrect?

• @Xi'an This is from a lecture by Gharamani but I've seen related statements by Gelman, MacKay etc – questionmark May 14 at 10:45

When doing model selection, each model $$\mathfrak M$$ is given an evidence $$\mathfrak e(\mathfrak M)$$ that writes as the corresponding integrated likelihood $$\mathfrak e(\mathfrak M) = \int f_{\mathfrak M}(\mathbf x|\theta_{\mathfrak M})\,\text{d}\theta_{\mathfrak M}$$Each model is then given a posterior probability $$\pi(\mathfrak M|\mathbf x)$$. The decision to select a model, if need be, is based on the maximisation of a utility function $$\arg\max_{\mathfrak M} \mathbb E[U(\mathfrak M,\theta_{\mathfrak M})|\mathbf x]$$Hence, it is correct that in the decision, optimisation occurs.