Is there any “maximizing” in a Bayesian model?

Question

I am not a statistician, so please bear that in mind. Suppose you have a simple model: $$ y_i \sim Normal(\mu, \sigma) \quad i = 1 \ldots n \\ \mu \sim f(\cdot) $$ where $f$ is some distribution for $\mu$ (and assume its not a conjugate distribution). The posterior distribution of $\mu$ can be found using Bayes Theorem, i.e. $$ p(\mu | y_1, y_2, \ldots, y_n) \propto p(y_1, y_2, \ldots, y_n | \mu) f(\mu) $$ where the right hand side product terms are the likelyhood and the prior. To find the posterior distribution, one can make use of an MCMC algorithm, for instance the Metropolis Hastings algorithm. In this algorithm, a Markov Chain is constructed so that the stationary distribution is the posterior distribution, and samples can be drawn using this MC.

My question is at which point does the observed data $y = (y_1^*, y_2^*, \ldots, y_n^* )$ get used to accurately predict the posterior? I was thinking something along the lines of sampling from the posterior distribution from $\mu$, using this drawn sample to sample $y_i$ and comparing against the observed value $y_i^*$. For example, in this JAGS pseudocode:

for i = 1:n
   y[i] ~ dnorm(mu, sigma=10)
end
mu ~ dnorm(0, 1)

Here the y (the data) gets sampled from the Normal distribution, but no comparison is made against the observed value.

My interpretation: I am guessing that the observed values are used only once in determining the formula for the posterior distribution. After the expression for the posterior is derived, MCMC takes over and data is no longer used. I guess I am a little shocked that the likelihood function can contain so much information to yield the posterior this way.

Answer 1

There is no maximization of anything unless you are using maximum a posteriori (MAP) estimation, where you maximize the posterior probability

$$ \hat \mu = \underset{\mu}{\operatorname{arg\,max}} \; p(\mu | y_1, y_2, \ldots, y_n) $$

Otherwise, when using MCMC, we are just taking samples from the posterior distribution. We don't need to "maximize" anything, since we know the posterior distribution by applying Bayes theorem, i.e. multiplying likelihood by the prior

$$ p(\mu | y_1, y_2, \ldots, y_n) \propto p(y_1, y_2, \ldots, y_n | \mu) f(\mu) $$

What we don't know, is the normalizing constant, that would enable us to get rid of the "$\propto$" sign and make this a proper conditional distribution. Hopefully, for optimization (MAP), or MCMC sampling, we don't need to know the normalizing constant.

So in a sense, yes, MCMC "takes" the posterior distribution and draws samples from it. However it is not correct to say that beforehand any "maximization" has taken place, since the only thing that has happened is we derived the posterior from likelihood and prior by multiplying one by another.