Basically, I have a multinomial distribution with probabilities $\theta_1, \theta_2 ... \theta_m$, and a Dirichlet prior (arbitrarily set as $\alpha_i = \alpha_{i+1} ... = 1$ (i.e. every $\alpha = 1$), which leads to a posterior distribution (which is also Dirichlet?)

As far as I know, the posterior distribution reflects a joint distribution of the updated $\theta$ values in light of the data, right? In doing so, I think the marginal distribution of, say, $\theta_1$, is obtained by a Gibbs sampler (or some suitable MCMC method)?

  • In this case, I know that the full conditional is supposed to be used, but what is the full conditional distribution for a Dirichlet posterior? Is it yet another Dirichlet that I sample from, where I sample for $\theta_1$ by fixing all my other $(m-1)$ $\theta$ values?
  1. The Dirichlet distribution is conjugate prior to the categorical and multinational outcomes. So the posterior distribution in the case of a Dirichlet prior with a multinational or categorical likelihood is also Dirichlet.

  2. Posteriors are priors that have been updated in light of the data. This is called Bayesian updating. Because of the conjugacy property, you don't need to worry about using sampling methods to obtain posteriors, provided you're using a conjugate model.

I'm not clear on what you mean by "full conditional." For a multinomial or categorical model with Dirichlet priors, the posterior distribution is a Dirichlet distribution, where each parameter value is the sum of the corresponding prior values and the observed values from your data. So once you've observed all of your data, the posterior distribution is fixed.

The wikipedia article, particularly the section on conjugate properties, is useful in answering this question.

  • $\begingroup$ Thanks for clearing this up @user777 -- I thought sampling was necessary but with conjugacy I guess that's unnecessary. Then for the new (posterior) Dirichlet, suppose I wanted to obtain the density for $\theta_1$ (since a random sample from the posterior is a vector [$\theta_1, \theta_2, \ldots \theta_m$, right?], then is it a matter of sampling from the posterior $n$ number of times, and then calculating the density of $\theta_1$? $\endgroup$ Nov 15 '15 at 18:53
  • $\begingroup$ There is no density of $\theta_1$, just your posterior. But perhaps you're interested in how the posterior distribution "looks." Point of interest: the Dirichlet distribution is just the multivariate extension of the beta distribution (with parallel conjugacy properties). So if you're only interested in the distribution of $\theta_1$, then you can collapse all of your other data points into a category corresponding to the negation of the category for $\theta_1$. Then you have just a beta distribution. $\endgroup$
    – Sycorax
    Nov 15 '15 at 19:09
  • $\begingroup$ Hm, okay (sorry for being totally dumb), I'm actually interested in all my $\theta$ values, and visualise the uncertainty of each of these θ's. What would be the best way to go about doing this with the new posterior? $\endgroup$ Nov 16 '15 at 0:33
  • $\begingroup$ As an FYI, I don't have to visualise all of the $m$ $\theta$ values, but maybe two of these as a toy example, e.g. $\theta_2, \theta_4$. Simply, what would be the best way to visualise my posterior? $\endgroup$ Nov 16 '15 at 0:50
  • $\begingroup$ In the Dirichlet-multinomial model, the values $\theta_\cdot$ are known precisely; there's no uncertainty there. But there is uncertainty in the probability of each outcome; in fact, the posterior is the distribution over that probability. Perhaps that's what you're interested in knowing about? There aren't great ways to visualize this, to my knowledge. The wikipedia page displays a visualization over trinomial outcomes, showing how much the density is concentrated near each corner. $\endgroup$
    – Sycorax
    Nov 16 '15 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.