3
$\begingroup$

I've been getting to grips with some Bayesian modelling, but one thing is confusing the heck out of me when I look at tutorials and worked-through problems online. I'm looking at a problem with a Dirichlet prior with alpha length 3, and observed data with a multinomial distribution. So we end up with a Dirichlet posterior. All the examples I look at online use MCMC sampling methods to form the posterior, but it is my understanding that you don't need to sample with conjugate distributions, since the posterior can be solved analytically.

This is an example of a tutorial that does this in pymc3

If I'm incorrect, and you do need to sample using conjugate priors, what is happening during each sampling step? Is it sampling one value from the Dirichlet distribution or three? Is it updating the Dirichlet prior with each step?

Thanks for any help.

Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ In this pedagogical example, the posterior distribution is a Dirichlet distribution and hence does not require MCMC. It is however a setting where you can illustrate MCMC with a comparison with the truth. There are however complex enough conjugate priors for which you need simulation, as for eg the Beta distribution.. $\endgroup$
    – Xi'an
    Apr 2, 2020 at 3:53

1 Answer 1

Reset to default
7
$\begingroup$

You are correct that if you have a conjugate prior, there's no need to use MCMC as the posterior has a closed form solution. MCMC tutorials that present a problem where we know the posterior already do so as nothing more than a demonstration, very similar to simulating some data from a given set of parameters and then demonstrating that our MLE estimate is fairly close to those values.

In practice, the set of models that are conjugate is much, much smaller than the set of non-conjugate models. Since those models do not have a closed form solution, we often use MCMC to approximate them.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In the case where we have conjugacy, we use a Gibbs sampler in the MCMC algorithm, correct? $\endgroup$
    – Ron Snow
    Apr 14, 2020 at 17:23
  • 1
    $\begingroup$ @Edison: in the case of complete conjugacy, we don't need MCMC at all, as the full posterior is in closed form. In the case where we have conjugacy for a subset of parameters, we may use Gibbs sampling for those conjugate parameters, although it's not always clear that there will benefits from doing so. $\endgroup$
    – Cliff AB
    Apr 14, 2020 at 18:14
  • 1
    $\begingroup$ Is complete conjugacy when every parameter is a conjugate prior? This is good to note. Thanks! $\endgroup$
    – Ron Snow
    Apr 14, 2020 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.