In LDA topic model algorithm, I saw this assumption. But I don't know why chose Dirichlet distribution? I don't know if we can use Uniform distribution over Multinomial as a pair?
$\begingroup$
$\endgroup$
2
-
7$\begingroup$ The uniform distribution is a special case of the dirichlet distribution. $\endgroup$– Stumpy Joe PeteNov 27, 2012 at 8:22
-
1$\begingroup$ @Xi'an's answer (below) helped me - clarifying that the Dirichlet distribution is A prior for the multinomial, not THE prior. It's chosen because it is a conjugate prior that works well to describe certain systems such as documents in NLP. $\endgroup$– Dan NissenbaumJul 20, 2021 at 2:11
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to compute and (b) it in some sense is possible to quantify how much our beliefs have changed after collecting the data.
It can certainly be discussed whether these are good reasons to choose a particular prior, as these criteria are unrelated to actual prior beliefs... Nevertheless, conjugate priors are popular, as they often are reasonably flexible and convenient to use for the reasons stated above.
For the special case of the multinomial distribution, let $(p_1,\ldots,p_k)$ be the vector of multinomial parameters (i.e. the probabilities for the different categories). If $$(p_1,\ldots,p_k)\sim \mbox{Dirichlet}(\alpha_1,\ldots,\alpha_k)$$ prior to collecting the data, then, given observations $(x_1,\ldots,x_k)$ in the different categories, $$(p_1,\ldots,p_k)\Big|(x_1,\ldots,x_k)\sim \mbox{Dirichlet}(\alpha_1+x_1,\ldots,\alpha_k+x_k).$$
The uniform distribution is actually a special case of the Dirichlet distribution, corresponding to the case $\alpha_1=\alpha_2=\cdots=\alpha_k=1$. So is the least-informative Jeffreys prior, for which $\alpha_1=\cdots=\alpha_k=1/2$. The fact that the Dirichlet class includes these natural "non-informative" priors is another reason for using it.
-
1$\begingroup$ So we choose Dirichlet distribution for those benefits. $\endgroup$ Nov 27, 2012 at 8:35
-
2$\begingroup$ +1: You may want to explicitly say that the likelihood is necessarily Dirichlet, which is why the posterior distribution is easy to compute. $\endgroup$– Neil GDec 16, 2012 at 19:51
$\begingroup$
$\endgroup$
In addition rather than contradiction to Måns T's answer, I simply point out that there is no such thing as "the prior" in Bayesian modelling! The Dirichlet distribution is a convenient choice because of (a) conjugacy, (b) computing, and (c) connection with non-parametric statistics (since this is the discretised version of the Dirichlet process).
However, (i) whatever prior you put on the weights of the multinomial is a legitimate answer at the subjective Bayes level and (ii) in case of prior information being available there is no reason it simplifies into a Dirichlet distribution. Note also that mixtures and convolutions of Dirichlet distributions can be used as priors.
