I am interested in finding clusters of words / topics in text. I am trying to learn more about potential approaches. The Wikipedia page on document clustering seems to provide a helpful overview (although I am sure there are limitations to this article and would be more than happy to be pointed to other resources).

In a research project, I've been using a two-step clustering approach (hierarchical and then k-means) with some degree of success but am interested in better understanding the landscape.

The Wikipedia article referenced above distinguishes between 'hard' and 'soft' approaches. According to it, hard approaches, such as hierarchical approaches and k-means, assign documents to a single cluster, while soft clustering approaches (the article says that both Latent Drichlet Allocation and topic models are examples of this approach) assign a mixture of clusters to a document. As a caveat, I'm not sure how Latent Drichlet Allocation and topic models can be considered clustering approaches (or whether they are), but my question is, what are the benefits to 'hard' and 'soft' approaches?

In short:

  • What are the benefits of using hierarchical and / or k-means clustering algorithms to identify clusters of words / topics in text?
  • What are the benefits of using Latent Drichlet Allocation / topic models to identify clusters of words / topics in text?
  • What do you mean with "clusters of words in text"? It sounds more like identifying frequent n-grams than topic modelling. – jknappen Aug 24 '15 at 12:36
  • You're right. I meant something more along the lines of "clusters of documents" and the most frequent words that appear in those documents, with the overall goal of grouping documents on the basis of their similarity and trying to determine why they are similar. – Joshua Rosenberg Aug 24 '15 at 12:39
up vote 2 down vote accepted

A classical clustering algorithm (like k-means or hierarchical clustering) gives you one label per document.

Topic modeling gives you a probabilistic composition of the document (so a document has a set of weighted labels). In addition, it gives you topics that are probability distributions over words.

Note that both procedures are unsupervised learning and far from being perfect, no matter how impressing the results may look at first sight. Apply them to dataset you understand well first!

  • I see. What if a clustering algorithm is combined with something like a cosine metric to compare a document to those in each cluster (e.g., something like Latent Semantic Analysis)? – Joshua Rosenberg Aug 24 '15 at 12:56
  • @Joshua Rosenberg Most clustering algorithms require some kind of metric space, otherwise you cannot use them. Usually one uses Euclidean distance (aka cosine metric) for that. – jknappen Aug 24 '15 at 13:03
  • Got it. What about when a metric space is used to compare one document to the mean word frequencies for the documents in a cluster? – Joshua Rosenberg Aug 24 '15 at 13:24

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.