Bayesian methods are about averaging over uncertainty rather than optimization. Explain? I came across the statement "The  key  ingredient  in  Bayesian  methods  is  to  average  over  your  uncertain variables and parameters, rather than to optimize". Can someone explain why this is? 
My confusion is that in Bayesian model selection, if you compute the model evidence, surely that involves some element of optimization... you then compute the posterior probability to get the Bayes factor, for example, but it seems there is some element of optimization still. Is this incorrect? 
 A: When doing model selection, each model $\mathfrak M$ is given an evidence $\mathfrak e(\mathfrak M)$ that writes as the corresponding integrated likelihood
$$\mathfrak e(\mathfrak M) = \int f_{\mathfrak M}(\mathbf x|\theta_{\mathfrak M})\,\text{d}\theta_{\mathfrak M}$$Each model is then given a posterior probability $\pi(\mathfrak M|\mathbf x)$. The decision to select a model, if need be, is based on the maximisation of a utility function
$$\arg\max_{\mathfrak M} \mathbb E[U(\mathfrak M,\theta_{\mathfrak M})|\mathbf x]$$Hence, it is correct that in the decision, optimisation occurs.
A: I did a cursory search for the context of the quote you give. I find it showing up a lot with slides for Gharamani. 
In those slides he's using that statement to suggest that MAP is not a bayesian method. My experience with Bayesian methods has been that the results of some inference are usually a distribution of parameters that you then average over in some way to get an estimation/prediction.
I think Xi'an demonstrates that getting the evidence itself doesn't require optimization. The only point where optimization comes into play is when you have to make a decision to select a model based on some utility function. 
