Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In LDA topic model algorithm, I saw this assumption. But I don't know why chose Dirichlet distribution? I don't know if we can use Uniform distribution over Multinational as a pair?

share|improve this question
2  
The uniform distribution is a special case of the dirichlet distribution. –  Stumpy Joe Pete Nov 27 '12 at 8:22
add comment

2 Answers 2

up vote 14 down vote accepted

The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to compute and (b) it in some sense is possible to quantify how much our beliefs have changed after collecting the data.

It can certainly be discussed whether these are good reasons to choose a particular prior, as these criteria are unrelated to actual prior beliefs... Nevertheless, conjugate priors are popular, as they often are reasonably flexible and convenient to use for the reasons stated above.

For the special case of the multinomial distribution, let $(p_1,\ldots,p_k)$ be the vector of multinomial parameters (i.e. the probabilities for the different categories). If $$(p_1,\ldots,p_k)\sim \mbox{Dirichlet}(\alpha_1,\ldots,\alpha_k)$$ prior to collecting the data, then, given observations $(x_1,\ldots,x_k)$ in the different categories, $$(p_1,\ldots,p_k)\Big|(x_1,\ldots,x_k)\sim \mbox{Dirichlet}(\alpha_1+x_1,\ldots,\alpha_k+x_k).$$

The uniform distribution is actually a special case of the Dirichlet distribution, corresponding to the case $\alpha_1=\alpha_2=\cdots=\alpha_k=1$. So is the least-informative Jeffreys prior, for which $\alpha_1=\cdots=\alpha_k=1/2$. The fact that the Dirichlet class includes these natural "non-informative" priors is another reason for using it.

share|improve this answer
    
So we choose Dirichlet distribution for those benefits. –  ColinBinWang Nov 27 '12 at 8:35
    
Great, thank you. :) –  ColinBinWang Nov 27 '12 at 8:46
    
+1: You may want to explicitly say that the likelihood is necessarily Dirichlet, which is why the posterior distribution is easy to compute. –  Neil G Dec 16 '12 at 19:51
add comment

In addition rather than contradiction to Måns T's answer, I simply point out that there is no such thing as "the prior" in Bayesian modelling! The Dirichlet distribution is a convenient choice because of (a) conjugacy, (b) computing, and (c) connection with non-parametric statistics (since this is the discretised version of the Dirichlet process).

However, (i) whatever prior you put on the weights of the multinomial is a legitimate answer at the subjective Bayes level and (ii) in case of prior information being available there is no reason it simplifies into a Dirichlet distribution. Note also that mixtures and convolutions of Dirichlet distributions can be used as priors.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.