I am not a statistician, so please bear that in mind. Suppose you have a simple model: $$ y_i \sim Normal(\mu, \sigma) \quad i = 1 \ldots n \\ \mu \sim f(\cdot) $$ where $f$ is some distribution for $\mu$ (and assume its not a conjugate distribution). The posterior distribution of $\mu$ can be found using Bayes Theorem, i.e. $$ p(\mu | y_1, y_2, \ldots, y_n) \propto p(y_1, y_2, \ldots, y_n | \mu) f(\mu) $$ where the right hand side product terms are the likelyhood and the prior. To find the posterior distribution, one can make use of an MCMC algorithm, for instance the Metropolis Hastings algorithm. In this algorithm, a Markov Chain is constructed so that the stationary distribution is the posterior distribution, and samples can be drawn using this MC.

My question is at which point does the observed data $y = (y_1^*, y_2^*, \ldots, y_n^* )$ get used to accurately predict the posterior? I was thinking something along the lines of sampling from the posterior distribution from $\mu$, using this drawn sample to sample $y_i$ and comparing against the observed value $y_i^*$. For example, in this JAGS pseudocode:

for i = 1:n
   y[i] ~ dnorm(mu, sigma=10)
mu ~ dnorm(0, 1)

Here the y (the data) gets sampled from the Normal distribution, but no comparison is made against the observed value.

My interpretation: I am guessing that the observed values are used only once in determining the formula for the posterior distribution. After the expression for the posterior is derived, MCMC takes over and data is no longer used. I guess I am a little shocked that the likelihood function can contain so much information to yield the posterior this way.

  • $\begingroup$ I am not sure to have well understood the question. From what I got, you want to compare the distribution of the sample obtained from the pseudocode (in which you simulate several values of $\mu$ right, for each of them you sample $n$ values of $y$, is this it?) to the empirical distribution of your sample $y^*$. These two distributions are not the same. Distribution of data $y^*$ is conditional to a (unknown) $\mu$ value, whereas the distribution you described is marginalized on $\mu$. $\endgroup$
    – Pohoua
    Jul 1 '20 at 18:27

There is no maximization of anything unless you are using maximum a posteriori (MAP) estimation, where you maximize the posterior probability

$$ \hat \mu = \underset{\mu}{\operatorname{arg\,max}} \; p(\mu | y_1, y_2, \ldots, y_n) $$

Otherwise, when using MCMC, we are just taking samples from the posterior distribution. We don't need to "maximize" anything, since we know the posterior distribution by applying Bayes theorem, i.e. multiplying likelihood by the prior

$$ p(\mu | y_1, y_2, \ldots, y_n) \propto p(y_1, y_2, \ldots, y_n | \mu) f(\mu) $$

What we don't know, is the normalizing constant, that would enable us to get rid of the "$\propto$" sign and make this a proper conditional distribution. Hopefully, for optimization (MAP), or MCMC sampling, we don't need to know the normalizing constant.

So in a sense, yes, MCMC "takes" the posterior distribution and draws samples from it. However it is not correct to say that beforehand any "maximization" has taken place, since the only thing that has happened is we derived the posterior from likelihood and prior by multiplying one by another.

  • $\begingroup$ So it correct to say that, if the prior was uninformative, then the posterior is "driven/derived" primarily from the data? $\endgroup$
    – masfenix
    May 31 at 1:57
  • $\begingroup$ @masfenix posterior depends on data and a prior, prior always has some effect on the result, “uninformative prior” is a misleading name stats.stackexchange.com/q/20520/35989 $\endgroup$
    – Tim
    May 31 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.