I have a corpus of documents. I want to do clustering of similar documents by using community discovery algorithm. Initially I preprocessed the corpus by using nltk. Then each document is converted into a vector(BOW approach).

Now pairwise cosine-similarity of all the documents is taken as edge weight between two nodes (node here is a single document ). On this graph I want to apply some community detection algorithm. Can anyone help me out to choose the good community detection algorithm for doing clustering. Is this approach okay?

  • $\begingroup$ Community detection isn't that different from regular clustering. Which also often uses cosine matrices. Make sure you know the literature well enough, e.g., HAC, spectral, etc. $\endgroup$ – Has QUIT--Anony-Mousse Mar 18 '19 at 19:07
  • $\begingroup$ Actually Sir, here in this community detection method I want to do clustering based on the weighted graphs but in regular clustering algorithms we do on the data set itself. Like in Girvan Newman algorithm edge betweenness is the center point. So which community detection algorithm is fast enough to do the partitioning of graph?? $\endgroup$ – anniebeniwal Mar 19 '19 at 9:44
  • $\begingroup$ And I studied clustering algorithms too but my project is on text document clustering by using community discovery algorithms. $\endgroup$ – anniebeniwal Mar 19 '19 at 9:50
  • 1
    $\begingroup$ Why go to the artificial "community" level in the first place? You have document similarities, not links here. Just cluster with a clustering algorithm, since most "community detection" works by clustering the nodes... $\endgroup$ – Has QUIT--Anony-Mousse Mar 19 '19 at 17:26
  • $\begingroup$ We need to go to community level because this is the work assigned to us by out prof.. Documents are taken as nodes and links between these nodes are created by using networkx library (Whole graph is created by using this library) . Where weights of the links are the cosine similarity between the nodes of that link. I want to work on this because may be this new algorithm will be having lesser time complexity and high accuracy compare to the traditional clustering approach. $\endgroup$ – anniebeniwal Mar 20 '19 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.