5
$\begingroup$

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is?

My confusion is that in Bayesian model selection, if you compute the model evidence, surely that involves some element of optimization... you then compute the posterior probability to get the Bayes factor, for example, but it seems there is some element of optimization still. Is this incorrect?

$\endgroup$
  • $\begingroup$ @Xi'an This is from a lecture by Gharamani but I've seen related statements by Gelman, MacKay etc $\endgroup$ – questionmark May 14 at 10:45
4
$\begingroup$

When doing model selection, each model $\mathfrak M$ is given an evidence $\mathfrak e(\mathfrak M)$ that writes as the corresponding integrated likelihood $$\mathfrak e(\mathfrak M) = \int f_{\mathfrak M}(\mathbf x|\theta_{\mathfrak M})\,\text{d}\theta_{\mathfrak M}$$Each model is then given a posterior probability $\pi(\mathfrak M|\mathbf x)$. The decision to select a model, if need be, is based on the maximisation of a utility function $$\arg\max_{\mathfrak M} \mathbb E[U(\mathfrak M,\theta_{\mathfrak M})|\mathbf x]$$Hence, it is correct that in the decision, optimisation occurs.

$\endgroup$
3
$\begingroup$

I did a cursory search for the context of the quote you give. I find it showing up a lot with slides for Gharamani.

In those slides he's using that statement to suggest that MAP is not a bayesian method. My experience with Bayesian methods has been that the results of some inference are usually a distribution of parameters that you then average over in some way to get an estimation/prediction.

I think Xi'an demonstrates that getting the evidence itself doesn't require optimization. The only point where optimization comes into play is when you have to make a decision to select a model based on some utility function.

$\endgroup$
  • $\begingroup$ I see, "The only point where optimization comes into play is when you have to make a decision to select a model based on some utility function." Are you referring to Bayes estimators/Bayes risk? $\endgroup$ – questionmark May 14 at 20:26
  • 1
    $\begingroup$ I'm more referring to the context that Xi'an is putting things in, which I think would be the estimators/risk context. $\endgroup$ – Kitter Catter May 14 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.