I am confused between what types of problems these three models capture, and their applications:

More than formal definitions (which I can find on Wikipedia), I am looking for the intuition behind each of them ( how they build on each other).

Perhaps, more specifically:

  • Is the difference between LDA and HDP that LDA is parameteric (i.e. I need to pre-specify the number of topics) whereas HDP is non-parametric? (and therefore I don't need to know how many topics I have)
  • What is the difference between a Dirichlet Processes and Pitman-Yor Processes?
  • What is the difference between a non-parametric process (e.g. DP) and a hierarhical non-parametric process (e.g. HDP)?
  1. Yes, when using the HDP approach to topic modeling, the number of topics is inferred from the data, whereas with standard LDA the number of topics must be pre-specified.
  2. The Pitman-Yor process is a simple generalization of the Dirichlet process which adds another parameter to the DP definition. Both processes are used as priors on cluster models where the number of clusters is unknown. Practically speaking, the Pitman Yor process allows more flexibility in the distribution of weights associated with each cluster (that is, the proportion of data points generated by each cluster). In a draw from the dirichlet process, if you put the cluster weights in decreasing order, they tend to decay exponentially. This is inconsistent with the clustering pattern observed in some domains such as language, hence the prominent use of Pitman Yor process priors in computational linguistics.
  3. The DP is used as a prior for clustered data. The HDP is used as a prior on grouped data where clusters are shared across groups. For example, in topic modeling, different documents are "groups" of data that share common topics (clusters). A single level DP cannot be used as a prior on topic models.

The Dirichlet Process (DP) is a stochastic process used in Bayesian non-parametric models. Each draw from a DP is a discrete distribution. DP exhibits a clustering property which makes it useful in mixture modeling where the number of mixture components grows with data.

Pitman-Yor (PY) process is an extension of DP that enables modeling of tail behavior with greater flexibility. It includes a discount parameter that controls the probability of cluster assignment. This makes PY useful for modelling data with power-law tails (such as word frequencies in natural language).

Hierarchical Dirichlet Process (HDP) is an extension of a DP that models problems involving groups of data, where each observation within a group is a draw from a mixture model and mixture components are shared between groups. In each group, the number of components is learned from data using a DP prior. HDP processes are useful in for example learning the topics shared across multiple corpora.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.