# Tag Info

11

The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to ...

8

In addition rather than contradiction to Måns T's answer, I simply point out that there is no such thing as "the prior" in Bayesian modelling! The Dirichlet distribution is a convenient choice because of (a) conjugacy, (b) computing, and (c) connection with non-parametric statistics (since this is the discretised version of the Dirichlet process). However, ...

6

In answer to your specific questions: The initial choice of prior was arbitrary. It was probably an attempt to produce an uniformative prior while avoiding an improper prior. Because a Dirichlet distribution is a conjugate prior with the property that to "update the Dirichlet distribution, we add the number of observations to each parameter", you just ...

6

I don't think this is an "overparamaterized" model at all. I would argue that by placing a prior over the Dirichlet paramaters, you're being less committal about any particular outcome. In particular, as you probably know, for symmetric dirichlet distributions (i.e. $\alpha_1 = \alpha_2 = ... \alpha_K$) setting $\alpha<1$ gives more prior probability to ...

6

This is a gentle tutorial: http://www.cs.cmu.edu/~kbe/dp_tutorial.pdf I like the wikipedia entry as well. The links there are also very good: http://en.wikipedia.org/wiki/Dirichlet_process Here is a summer school lecture of one of the most active researchers http://videolectures.net/mlss07_teh_dp/

5

As far as I know you just need to supply a number of topics and the corpus. No need to specify a candidate topic set, though one can be used, as you can see in the example starting at the bottom of page 15 of Grun and Hornik (2011). A relatively simple way to find the optimum number of topics without training data is by looping through models with ...

5

One way to have a random $\theta=(\theta_1,\dots,\theta_k)$ living on the simplex, without the limitations imposed by the negative covariances of the Dirichlet distribution, is to define $\phi_i=\sum_{j=1}^k c_{ij} \log \theta_j$, for $i=1,\dots,k-1$, where the $(k-1)\times k$ matrix $C=(c_{ij})$ has rank $k-1$. Adding the constraint ...

4

The error appears to be caused by the fact that x is not a matrix. The reason why x needs to be a matrix, in principle is that the Dirichlet is a multivariate distribution; each of your $N$ observations has, let us say, $K$ elements, leading naturally to a representation of x as an $N \times K$ matrix. Even if you only have one observation, the function as ...

4

Remember that, if $Y_i$ are independent $\mathrm{Gamma}(a_i,b)$, for $i=1,\dots,k$, then $$(X_1,\dots,X_k) = \left(\frac{Y_1}{\sum_{j=1}^k Y_j}, \dots, \frac{Y_k}{\sum_{j=1}^k Y_j} \right) \sim \mathrm{Dirichlet}(a_1,\dots,a_k) \, .$$ The proof can be found on page 594 of Luc Devroye's book. Therefore, one possibility is to compute a Monte Carlo ...

4

A Beta distribution is just a special case of the Dirichlet distribution, that is, a Beta distribution is a Dirichlet distribution with two parameters, alpha and beta. Dirichlet is the multidimensional generalisation (of Beta) with 'n' parameters instead of two. The parameters of Dirichlet are denoted by alpha with an index as a subscript. Setting all the ...

3

You suggest in comments that you don't know how the log-gamma function will work with the likelihood. You need some basic facts: 1) log of a product of terms is the sum of the logs ($\log(ab) = \log(a) + \log(b)$) 2) log of a reciprocal is the negative of the log ($\log(1/c) = -\log(c)$) 3) The relationship between factorials and Gamma functions ($x! = ... 3 The keyword is compositional data. Here is a link to a course note with links to other papers/books explaining the basic characteristic/challenges associated with this type of data (in particular for a textbook like reference see (1)) (1) Aitchison, J., 1986. The Statistical Analysis of Compositional Data. 3 First, you need to put the data into a sensible form for ggplot2: dat <- data.frame(item=factor(rep(1:10,15)), draw=factor(rep(1:15,each=10)), value=as.vector(t(x))) Then you can plot it by building up the components you can see in the plot (points and lineranges; faceting, axis control and facet borders): ... 3 For me the most helpful way to envision the effect of the parameters for Dirichlet is the Polya urn. Imagine you have an urn containing n different colors, with$\alpha_i$of each color in the urn (note that you can have fractions of a ball). You reach in and draw a ball, then replace it along with another of the same color. You then repeat this an infinite ... 3 You should be able to use KDR to estimate a Dirichlet type distribution, you just need to ensure you capture the constraints correctly. First off, drop the dimensionality by 1. As you correctly point out, in the 2D case, you have a beta distribution, hence there is only 1 degree of freedom. In this case the correct thing to do would be to estimate the ... 3 Yes, when using the HDP approach to topic modeling, the number of topics is inferred from the data, whereas with standard LDA the number of topics must be pre-specified. The Pitman-Yor process is a simple generalization of the Dirichlet process which adds another parameter to the DP definition. Both processes are used as priors on cluster models where the ... 2 The mean and variance of a Gaussian are the unknown parameters that specify that distribution in that case. Likewise, in topic modeling, you attempt to learn the unknown parameters of$K$topics, where each topic is a multinomial distribution over words in the vocabulary. Thus, it is the parameters of each multinomial distribution (each topic) that you seek ... 2 This is a direct Bayesian conjugate prior modeling. A natural extension from Beta-Binomial model. A good resource for this could be from the book. And Posterior is also Dirichlet and hence simulating from dirichlet will give necessary summaries 2 The p.d.f of the Dirichlet distribution is defined as $$f(\theta; \alpha) = B^{-1} \prod_{i=1}^K \theta_i^{\alpha_i - 1}$$ where$B(\alpha)$is the generalized Beta function. Notice that if any$\theta_i$is 0, then the whole product is zero. In other words, the support of a Dirichlet distribution is over vectors$\theta$where each$\theta_i \in (0, 1)$... 2 If$Y_i$are independent$\mathrm{Gamma}(\alpha_i,\beta)$, for$i=1,\dots,k$, then $$(X_1,\dots,X_k) = \left(\frac{Y_1}{\sum_{j=1}^k Y_j}, \dots, \frac{Y_k}{\sum_{j=1}^k Y_j} \right) \sim \mathrm{Dirichlet}(\alpha_1,\dots,\alpha_k) \, .$$ So, in R just do something like rdirichlet <- function(a) { y <- rgamma(length(a), a, 1) return(y / ... 2 1: denoting$z_{d,n} = A$,$w_{d,n} = B$,$\theta_{d} = C$,$\beta_k = D$, and using the Bayes rule:$\displaystyle p(A|B,C,D) = \frac{p(A|C,D) \ p(B|A,C,D)}{p(B|C,D)} \propto p(A|C,D)\ p(B|A,C,D)$, where$\propto$means it is proportional, which is the case because$p(B|C,D)$does not depend on$A$i.e. is a constant. 2: we can get knowing that both ... 2 You probably can do better than both of those. What people often do is average together the full conditional distributions they get while running their Markov chain. If you can do that - and you can, provided your Dirichlet update is conjugate - this is a better strategy. So, for example, suppose I want the posterior density of$\theta$, give the data. I ... 1 Do you mean equation (2)? I think he's not using Bayes' theorem at all -- he's just using the definition of conditional probability. Recall that if$A$,$B\$ are events, then $$P(A | B) = \frac{P(A \cap B)}{P(B)}.$$ If you want to know more about conditional probability, I think the Wiki article is pretty good.

1

I would suggest you look at page 8 of "Probabilistic Topic Models" by Mark Steyvers and Tom Griffiths. I found their explanation of the Gibbs algorithm quite clear and easy to implement. To answer your questions: i seems to range over (indexes for) all the words in all the documents, and d indeed seems to refer to the document of the word under ...

1

It sounds like your Markov chain can't travel among the hidden states because the modes on the probability surface are too far apart and/or separated by some very improbable states. For example, if your objects are vehicles and your two styles are "racecar" and "minivan", you'll have a lot of trouble getting from one to the other. Imagine trying to update ...

1

To calculate the density of any conjugate prior see here. However, you don't need to evaluate the conjugate prior of the Dirichlet in order to perform Bayesian estimation of its parameters. Just average the sufficient statistics of all the samples, which are the vectors of log-probabilities of the components of your observed categorical distribution ...

1

I wrote how to do this in my dissertation work. Minka's notes are actually quite helpful. If you take a look at the procedures to obtain the mle of the mean and precision parameters in an iteratice way, it can be modified slightly to obtain the mle under the null hypothesis that the means of two ind. Dirichlet are equal while having seperate precisions. ...

1

Its not a paper, but I found some Matlab code that implements a DP prior for an infinite Gaussian mixture model. The code uses Gibbs sampling to infer a GMM (and the number of components in the mixture) over some input data. The code is pretty readable and it has helped me quite a bit to see the DP in action in a concrete example.

1

I have the same feeling. This is as closest as I've come: http://www.cs.cmu.edu/~kbe/dp_tutorial.pdf The algorithm explained starting at page 37 is understandable, however I still wish to see a very simple example written out step by step so I'm more confident of which each term means. I attempted to implement the algorithm in R, below is my code, not ...

Only top voted, non community-wiki answers of a minimum length are eligible