1
$\begingroup$

Basically, I have a multinomial distribution with probabilities $\theta_1, \theta_2 ... \theta_m$, and a Dirichlet prior (arbitrarily set as $\alpha_i = \alpha_{i+1} ... = 1$ (i.e. every $\alpha = 1$), which leads to a posterior distribution (which is also Dirichlet?)

As far as I know, the posterior distribution reflects a joint distribution of the updated $\theta$ values in light of the data, right? In doing so, I think the marginal distribution of, say, $\theta_1$, is obtained by a Gibbs sampler (or some suitable MCMC method)?

  • In this case, I know that the full conditional is supposed to be used, but what is the full conditional distribution for a Dirichlet posterior? Is it yet another Dirichlet that I sample from, where I sample for $\theta_1$ by fixing all my other $(m-1)$ $\theta$ values?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

  1. The Dirichlet distribution is conjugate prior to the categorical and multinomial outcomes. So the posterior distribution in the case of a Dirichlet prior with a multinomial or categorical likelihood is also Dirichlet.

  2. Posteriors are priors that have been updated in light of the data. This is called Bayesian updating. Because of the conjugacy property, you don't need to worry about using sampling methods to obtain posteriors, provided you're using a conjugate model.

I'm not clear on what you mean by "full conditional." For a multinomial or categorical model with Dirichlet priors, the posterior distribution is a Dirichlet distribution, where each parameter value is the sum of the corresponding prior values and the observed values from your data. So once you've observed all of your data, the posterior distribution is fixed.

The wikipedia article, particularly the section on conjugate properties, is useful in answering this question.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks for clearing this up @user777 -- I thought sampling was necessary but with conjugacy I guess that's unnecessary. Then for the new (posterior) Dirichlet, suppose I wanted to obtain the density for $\theta_1$ (since a random sample from the posterior is a vector [$\theta_1, \theta_2, \ldots \theta_m$, right?], then is it a matter of sampling from the posterior $n$ number of times, and then calculating the density of $\theta_1$? $\endgroup$
    – thestatnoob
    Commented Nov 15, 2015 at 18:53
  • $\begingroup$ There is no density of $\theta_1$, just your posterior. But perhaps you're interested in how the posterior distribution "looks." Point of interest: the Dirichlet distribution is just the multivariate extension of the beta distribution (with parallel conjugacy properties). So if you're only interested in the distribution of $\theta_1$, then you can collapse all of your other data points into a category corresponding to the negation of the category for $\theta_1$. Then you have just a beta distribution. $\endgroup$
    – Sycorax ♦
    Commented Nov 15, 2015 at 19:09
  • $\begingroup$ Hm, okay (sorry for being totally dumb), I'm actually interested in all my $\theta$ values, and visualise the uncertainty of each of these θ's. What would be the best way to go about doing this with the new posterior? $\endgroup$
    – thestatnoob
    Commented Nov 16, 2015 at 0:33
  • $\begingroup$ As an FYI, I don't have to visualise all of the $m$ $\theta$ values, but maybe two of these as a toy example, e.g. $\theta_2, \theta_4$. Simply, what would be the best way to visualise my posterior? $\endgroup$
    – thestatnoob
    Commented Nov 16, 2015 at 0:50
  • 1
    $\begingroup$ If you sample from the posterior, those samples won't just have some distribution, but will be samples from the posterior itself! The Dirichlet is a distribution over the $(m-1)$ dimensional simplex, which is why it's used as a probability distribution over multinomial probabilities. Your approach makes sense. Hope this helps. $\endgroup$
    – Sycorax ♦
    Commented Nov 16, 2015 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.