I am studying LDA, but have very weak statistical knowledge. I have a question regarding Gibbs sampling, one of the methods for inferring the distribution of topics and words-topic given a document, which basically iterates and computes the probability of words from being assigned to each topic after removing the specific word from the counts. The question is, why it works ?
I found an explanation below, but I am not able to understand the parts in bold ...
Word probabilities are maximized by dividing the words among the topics. (More terms means more mass to be spread around.) In a mixture, this is enough to find clusters of co-occurring words. In LDA, the Dirichlet on the topic proportions can encourage sparsity, i.e., a document is penalized for using many topics. Loosely, this can be thought of as softening the strict definition of “co-occurrence” in a mixture model. This flexibility leads to sets of terms that more tightly co-occur.