4
$\begingroup$

I have a very rough understanding of the Dirichlet distribution and already seen some visualizations of its pdf over the 2-simplex, i.e., $\alpha$ is a 3D vector. However, I still do not understand the following statements that I have been frequently encountering with

Dirichlet distribution is a distribution over a multinomial distribution...

or

...draw a multinomial distribution from a Dirichlet distribution.

I would appreciate if someone could explain what these two statements mean, like to a non-statistian (a dummy).

$\endgroup$
5
$\begingroup$

Let's take a step back from the Dirichlet distribution and multinomial distribution and consider a slightly simpler set of models.

The binomial distribution describes the number of "successes" $y$ one expects to observe in a number of trials $n$. The binomial model has several key properties:

  • The binomial model is dichotomous.
  • Each event in a binomial model has a probability of "success" $\theta$.
  • The trials in a binomial model are independent: previous successes neither increase nor decrease the probability of future successes.

This is all well and good when all of our data are subject to some kind of rigorous controls, so that we know that across all of our observations, the $\theta$ for each "batch" of trials is the same. But more realistically, we have reason to believe that $\theta$ for trial $y_i$ is different than for trial $y_j$. One way to accomplish this is to treat each $\theta$ as if it were drawn separately from a beta distribution. The beta distribution has several useful properties

  • It is a probability distribution of probabilities: that is, it has support on the unit interval
  • It is conjugate to the binomial model, which simplifies computations (I have intentionally omitted an extended discussion of conjugacy in this post and how that can simplify the process of drawing values from these distributions because I feel that is only tangentially related to your question.)

One method to draw values from a beta-binomial model is the following set of steps:

  • Draw a value $\tilde{\theta}$ from the beta distribution.
  • Draw a value $y$ from the binomial distribution with $\theta=\tilde{\theta}$.

This covers the simple case of a dichotomous, binary outcome. But you've asked about Dirichlet distributions. Happily, the Dirichlet distribution is actually the same thing as a beta distribution when the dimension is 2. In higher dimensions, it is analogous to the beta distribution.

Likewise, the multinomial distribution is the higher-dimensional analogue to the binomial distribution. In the case of a dichotomous outcome, the binomial distribution is the multinational outcome.

Drawing values from a multinomial distribution with a Dirichlet distribution over the probabilities of outcomes is accomplished in a very similar way:

  • Draw a vector of probabilities from the Dirichlet distribution.
  • Use that vector of probabilities to draw a vector of outcomes from the multinomial distribution.
$\endgroup$
6
  • $\begingroup$ Thanks a lot for the bridge from Binomial+Beta to Multinomial+Dirichlet! That clears up things a lot! One follow-up question: you said "more realistically", $\theta$ varies. I agree. But how do we justify Beta distribution is a good distribution for us to draw $\theta$ from? Similarly, why do we draw vector of probability from Dirichlet other than some other distributions? Thanks a lot for the help! $\endgroup$ Sep 30 '14 at 15:46
  • $\begingroup$ In Bayesian inference, these distributions are convenient because they have the property of conjugacy. Also, because they are distributions of probabilities, they respect the constraint that probabilities must be positive and sum to 1 over MECE outcomes. But it's certainly possible to use appropriately-transformed alternative models, e.g. a (multinomial) logistic model. $\endgroup$
    – Sycorax
    Sep 30 '14 at 15:49
  • $\begingroup$ More specifically, I am wondering why we have latent Dirichlet allocation instead of latent SomeOtherDistribution allocation. So it seems that conjugacy is an unavoidable concept, if I wish to truly understand why we use Beta/Dirichlet. Alright, I can learn that. :) $\endgroup$ Sep 30 '14 at 15:52
  • $\begingroup$ I think that would make a good follow-on question. $\endgroup$
    – Sycorax
    Sep 30 '14 at 15:54
  • $\begingroup$ How do you sample from a multinomial distribution? For example in LDA, first a multinomial is sampled from a Dirichlet distribution. However, individual topics and words are taken from these multinomials. There is a missing link there, somehow. When you sample a multinomial you get a vector of values, right, and not just a single value? Individual vector entries must be sampled from the resulting multinomial vector, right? $\endgroup$ Nov 24 '16 at 11:51
0
$\begingroup$

I would appreciate if someone could explain what these two statements mean, like to a non-statistian (a dummy).

Let me try to do it by avoiding complex terminology.

First, you should understand what a simplex is. This answer explains it graphically: https://stats.stackexchange.com/a/296779/172825

To recap, every point in that simplex-triangle has a very nice property - it's in fact a particular multinomial distribution. Why? If you look at any point, its "vector" sums up to one and all "items" are between 0 and 1.

I tried to naively visualize Dirichlet distribution in the following figure.

figure

(a) shows a 3-simplex (as explained in the linked question). There are 3 example points shown with their "coordinates" so you can see that each of them is a multinomial distribution.

(b) let's put the triangle on the floor and the vertical axis be a probability density function

(c) is one particular example of Dirichlet distribution (I guess the parameters might be $\alpha = (2, 2, 2)$, roughly) - a kind of "mountain" in the middle of the triangle (sorry, I can't really sketch 3-D mountains easily). But you get the gist - the higher on that mountain, the higher the PDF value, thus the higher the chances a random sample will end-up here.

(d) just shows the "mountain" from the top as a contour-line plot; in reality the circles are not actual circles but that's a detail (e) is the same thing but using some kind of heat map (so the darker the blue is, the higher probability); such heat maps are shown in some literature on Dirichlet distribution

(f) is another Dirichlet distribution (different $\alpha$ parameters)

So what does it mean to "draw a multinomial distribution from a Dirichlet distribution"? You simply pick any point from that triangle (because every point is a multinomial distribution, remember?). Darker areas (or "higher" areas if you like the mountain analogy) just get higher chances to get sampled.

I'm not a statistician so I hope it's a correct explanation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.