The objective of my work is to cluster the text documents. Once the documents are clustered, traditionally the system will assign numeric value for the clustered group. For example if I have 5 categories, then the clustered documents are labeled by any one of these numerical values {1,2,3,4,5}. I would like to assign the cluster name (e.g., philosophy, biology,...) automatically rather than labeling it as {1,2,3,...} for further processing. My initial idea is to provide the cluster name by counting the highest frequency word in that cluster. I am confused if this approach is good or not. I am using k-means clustering. Currently I am excluding LDA (Latent Diriclet Allocation) or other methods.

  • If you know the names of the clusters before you start then why do cluster analysis (unsupervised learning) instead of some form of supervised learning? If you don't know the names before you start, then I don't see how you can automate the process. – Peter Flom Jun 23 '14 at 12:02
  • This question appears to me not statistical or analytic, but rather programming. The real relevant question here is whether K-means is adequate given that your data is seemingly categorical. – ttnphns Jun 23 '14 at 13:10
  • This is a text mining question and it is a very usual problem in the area. However few people would start from k-means (for scaling and flexibility reasons). LDA and methods with flexible number of clusters are more obvious choices. – iliasfl Jun 23 '14 at 13:43

Yes it makes sense, however frequencies of single words may lead to trivial topics. You need to do some kind of normalisation by using TF-IDF and finding the most informative word(s) in the cluster (IDFs should be computed on the whole corpus). In other words you want the most frequent word(s) in the cluster that are in-frequent in other clusters.

It would be better to go for sets of 2-3 consecutive words (easily discoverable by running some adaptation of a-priori on each cluster). Also search for meme tracking algorithms as an alternative option.

One technique for this is unsupervised multi-document keyword extraction. That is, extracting the most salient words and/or collocations for each of your clusters. A number of methods are available to do that, most of which are centered around either using graph centrality measures (think: Google's PageRank) or using language processing (NER/noun phrase detection) of each document, or simply using all possible n-grams and then extracting the most significant entities/phrases according to, e.g., their log(TF)-IDF scoring or log(TF)-Entropy score. Instead of using all n-grams, you could also detect collocations (ideally, with a likelihood ratios estimator, p. 161 ff.) on a large, independent corpus, and then use those (only) to form any higher n-gram.

However, in your case, maybe, statistical (feature) selection techniques might be more appropriate, like Boostrap testing to extract the terms (again, either sampling over all possible n-grams or by selecting specific, pre-computed collocations) that are significant for each cluster, relatively to all others. Note that you can use either the raw word counts or plug in any re-weighted count from the TF-IDF family of term weighting functions mentioned above. However, these statistical techniques require that you have access to all words (viz, word counts) among all clusters and indeed have the computational resources to do an all against all comparison, something that might be infeasible if your clusters and word data are too large.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.