10

The notion of "topics" in so-called "topic models" is misleading. The model does not know or is not designed to know semantically coherent "topics" at all. The "topics" are just distributions over tokens (words). In other words, the model just capture the high-order co-occurrence of terms. Whether these structures mean something or not is not the purpose of ...


10

What is the difference between (Dirichlet) distribution and (Dirichlet) process? The difference between a Dirichlet distribution and a Dirichlet process is perhaps easier to understand when you understand the difference between a Gaussian distribution and a Gaussian process. A Gaussian distribution pertains to the possible realizations of a single random ...


8

With certainty, realizations of a Dirichlet Process are probability measures with countable support, as proved by D. Blackwell, The Annals of Statistics 1 (1973), no. 2, 356--358. You can sample realizations from a Dirichlet Process using the constructive stick-breaking representation introduced by J. Sethuraman, Statistica Sinica, 4, 639 (1994). For a ...


7

For my own curiosity, I applied a clustering algorithm that I've been working on to this dataset. I've temporarily put-up the results here (choose the essays dataset). It seems like the problem is not the starting points or the algorithm, but the data. You can 'reasonably' (subjectively, in my limited experience) get good clusters even with 147 instances ...


7

I would recommend you to read Introduction to the Dirichlet Distribution and Related Processes by Bela A. Frigyik, Amol Kapila, and Maya R. Gupta. It is written in a very accessible manner, and I could imagine, it would be a great introduction to those topics (measure theory, etc) Hope it helps


5

What is the difference between DP and CRP? The Chinese Restaurant Process (CRP) is a distribution over partitions of integers. The connection to the Dirichlet Process (DP) exists thanks to De Finetti's theorem. De Finetti's theorem: Suppose we have a random process $(\theta_1,\ldots,\theta_N)$ that is infinitely exchangeable, then the joint probability $p(\...


5

Look at the densities. By virtue of the definition of the Dirichlet distribution, on the right hand side the density (in variables $\mathbf x = (x_1,x_2,\ldots,x_k)$ with $x_1+x_2+\cdots+x_k=1$) is proportional to $$F_\gamma(\mathbf x) = \frac{x_1^{\gamma_1-1}\cdots x_k^{\gamma_k-1}}{\Gamma(\gamma_1)\cdots\Gamma(\gamma_k)}.$$ On the left hand side, ...


5

To demonstrate a solution to this hyperprior problem, I implemented an hierarchical gamma-Dirichlet-multinomial model in PyMC3. The gamma prior for the Dirichlet is specified and sampled per Ted Dunning's blog post. The model I implemented can be found at this Gist but is also described below: This is a Bayesian hierarchical (pooling) model for movie ...


5

This is indeed rather confusing: the notation $d\phi$ stands for an infinitesimal measurable set located around $\phi$. As in standard measure theory settings with Leibniz's $dx$. It can thus be used in integrals as $$\mathbb{P}(\Phi_k^*\in A|B_{1:n},X_{1:n})=\int_A \mathbb{P}(\Phi_k^*\in d\phi|B_{1:n},X_{1:n})$$ to borrow from eqn (2.33) in Peter Orbanz' ...


5

Since $p(1)=p$ and $p(0)=1-p$ are both proportional to a known expression* (the unscaled probabilities, $u(i)=c.p(i)$, with the same unknown constant of proportionality, $c$) and you know the $p(i)$ values must add to $1$, then $u(0)+u(1)=c$. Which is to say $p(i) = \frac{u(i)}{u(0)+u(1)},\: i=0,1$. (This notion is widely used in Bayesian statistics with ...


5

The first slide applies the general result of the second slide to a case when $$y=\overbrace{(0,\ldots,0,1,0,\ldots,0)}^{1\text{ at }\kappa\text{th position}}$$ with different notations, since [in the second slide] $\alpha'_j=\alpha_j+n_j$ and $y^{(j)}=n_j\in\{0,1\}$. Then \begin{align*} \dfrac{\Gamma(n+1)}{\prod_{j=1}^K \Gamma(y^{(j)}+1)}&=\dfrac{\Gamma(...


4

Hopefully someone can give you a better answer, but I think the main point is this: the probabilities associated to each cluster decay, on average, exponentially for the Dirichlet process. Consider the stick breaking construction, where we let $\beta'_k \sim \mbox{Beta}(1, \alpha)$ and $\beta_k = \beta'_k \prod_{j < k} (1 - \beta'_j)$ where $\beta_k$ is ...


4

With probability one, the realizations of a Dirichlet Process are discrete probability measures. A rigorous proof can be found in Blackwell, D. (1973). "Discreteness of Ferguson Selections", The Annals of Statistics, 1(2): 356–358. The stick breaking representation of the Dirichlet Process makes this property transparent. Draw independent $B_i\sim\mathrm{...


4

You have two very different priors on $\pi$, so why would you expect the same posterior? Let's think about the nature of $T$ in your presentation above: In 1: Dirichlet Distribution, $T$ is a fundamental defining feature of the finite-dimensional Dirichlet prior distribution for $\mathbf{\pi}$. With $T$ fixed, you have complete flexibility (within the unit ...


4

You need to be very precise about what $c$ and $n$ are. Let $c_i$ be the number of customers at table $i$. Let $n$ be the customer number, i.e. $n-1$ clients are already seated at all tables. Let $k$ be the current number of tables. Then randomly choose a table using this probabilities: $$ P(T_i) = \begin{cases} c_i / (n - 1 + \alpha) & \text{ for }i\...


4

Denote by $\mathcal{M}$ a measurable space of probability measures, containing the realisations of the Dirichlet process. The random probability measure $G$ is a measurable function $$ G : \omega \mapsto G_\omega \in \mathcal{M} $$ and the integral with respect to $G$ is the random variable $$ \int f(\,\cdot\,|\, \psi) dG(\psi) : \omega \mapsto \int f(\,\...


4

The (conditional) marginal distribution can be computed from the (conditional) joint distribution: $$ p(\alpha|k) = \int p(\alpha,\eta|k)\,d\eta. $$ Therefore, if $$ p(\alpha,\eta|k) = c\,p(\alpha)\,\alpha^{k-1}\,(\alpha+n)\,\eta^{\alpha}\,(1-\eta)^{n-1}, $$ where $c$ is an appropriate constant, then $$ p(\alpha|k) = c\,p(\alpha)\,\alpha^{k-1}\,(\alpha+n)\,\...


4

Probably it is better to describe how one would generate data from a Dirichlet process mixture. Each line is understood to be conditional on all lines above it. Sample $P \sim \operatorname{DP}(\alpha P_0)$. Sample $\theta_1, \ldots, \theta_n \sim P$ independently. Sample $X_1, \ldots, X_n$ independently such that $X_i \sim f_{\theta_i}$. That's all ...


4

Chinese restaurant process Let's define this problem using the Chinese Restaurant Process (CRP) formulation of the Dirichlet Process (DP), which can be summarized as follows (from Gershman et al., emphasis mine): Imagine a restaurant with an infinite number of tables, and imagine a sequence of customers entering the restaurant and sitting down. The first ...


4

We can apply the Lyapounov CLT to show that $K_T$ is normal for large $T$. Let $$ K_T = \sum_{i=1}^T I(\text{new table at $i$}) = \sum_{i=1}^T Z_i. $$ Then it is well-known that $Z_i \stackrel{\text{ind}}{\sim} \text{Bernoulli}(p_i)$ where $p_i = \alpha / (\alpha + i - 1)$. Now, $\sum_{i=1}^\infty p_i (1 - p_i) = \sum_{i=1}^\infty \frac{\alpha i}{(\alpha + ...


3

You have to train your model, get the topics distribution for both the corpus you want to compare and then you need to choose a metric to compare them. For example, the topic distributions are vectors, and you can use the euclidian distance between them as an indicator of the difference between the documents. EDIT - (example) With gensim, you'll have to ...


3

Since you want a bayesian approach, you need to assume some prior knowledge about the thing you want to estimate. This will be in the form of a distribution. Now, there's the issue that this is now a distribution over distributions. However, this is no problem if you assume that the candidate distributions come from some parameterized class of distributions....


3

Check out the package DPackage in R. It has a lot of functionality for simulating from the Dirichlet Process. Here is a link to the documentation: DPackage. Zen's answer above is pretty good info as well.


3

which other clustering methods (unsupervised classification) can I try for this problem? For instance, parametric ones: you can fit a Gaussian Mixture Model by Expectation Maximization or Variational Bayes Inference; you test for different number of clusters and select the model that best fits your data. Be careful, model selection is not the same for a ...


3

A realization from a DP is a discrete distribution $G$, not a Dirichlet distribution. Basically a DP is a distribution from which you sample distributions. If you ever studied how you can sample Bernoulli or Binomial distributions from a Beta distribution, or Multinomial from a Dirichlet distributions this should not be totally strange. Basically, since the ...


3

The posterior base distribution must be a probability distribution. Let $M$ be the concentration parameter. For any $M$ $$\frac{\alpha}{M}H+\frac{1}{M}\sum_i \delta_{\theta_i}(\cdot)$$ is a finite measure, but it's only a probability measure if its total mass is 1, which will clearly be true for a unique $M$, and is true for $M=\alpha+n$.


3

The notes you reference have a splendid, clear, insightful discussion of this definition, its meaning, and its application--thank you for bringing them to our attention. Because the job of explanation has been so well done, all that is needed may be to clarify the specific points in your question. It looks like you lack only a definition of a "partition:" ...


2

Actually it was not "removed", the symbol does not mean equal! It means that the equation is proportional to the other. The denominator of the function is used to normalize the value and obtain a distribution in the range [0,1]. For this reason the two equations are proportional, and differ only for a scale factor.


2

These are two great tutorials, "Introduction to the Dirichlet Distribution and Related Processes" "A Very Gentle Note on the Construction of Dirichlet Process" specially the first one, with a reference to a very succinct tutorial on measure theory. I would start with the first one, because it starts by introducing the Dirichlet distribution and sampling, ...


2

1) What is the different between Chinese Restaurant Model and DP? None. CRP is a particular representation of DP. Depending on your problem you might want to use one representation over another (CRP, Stick-breaking, etc). 2) What is the different between Infinite Mixture Models and DP? DP is just used as a prior for the Infinite Mixture Model. This is why ...


Only top voted, non community-wiki answers of a minimum length are eligible