3
$\begingroup$

I have one million of keywords (from search queries in google), and I need to group them semantically. I have already done some research and I have found information about how to extract keywords and cluster them from a large corpus, but in my case I don't have any large documents, only those keywords.

I imagine that clustering these keywords semantically is impossible (although I hope I am wrong) since I guess you need a large text to extract its meaning, and each of my keywords has a maximum of 4 or 5 words. I thought about crawling the web and getting myself a large corpus and use some of the techniques I have seen like TF-IDF and then applying a k-means algorithm, and then I could extract the keywords from those documents and its subject, and then I could compare my keywords to those extracted and cluster them accordingly... But I don't know if this would work.

Could anyone tell me if my approach is correct? If so, once I have clustered the keywords from the documents I get from the web, what kinds of techniques would I need to cluster my own keywords?

Improve this question
$\endgroup$
2
  • $\begingroup$ Are those generic keywords or are they specific to a certain domain? In the latter case maybe specific databases do already exist! $\endgroup$ – nico Jan 7 '14 at 17:45
  • $\begingroup$ Unfortunately they are from almost every possible domain... $\endgroup$ – Balduz Jan 7 '14 at 18:36
2
$\begingroup$

I figure the best you can do is to try to map the keywords to Wordnet, and use the wordnet hierarchy as "clustering". You can also try to discover frequent itemsets.

You want to cluster them by their meaning, I guess. How is an algorithm that only has the characters available to discover a meaning here? IMHO you will need to provide any algorithm more data than just the words to produce a sensible result.

Improve this answer
$\endgroup$
3
  • $\begingroup$ True, I know that with only the keywords there is no possible way of figuring out its meaning. That is why I thought of crawling the web using Nutch or something similar, build a huge corpus so I can extract keywords and cluster them, and then finding similarities between my keywords and those from the web so I can use the clusters found from the web as clusters for my own set of keywords... I am new to this kind of things, my field is software engineering, but I think this could make sense... $\endgroup$ – Balduz Jan 8 '14 at 14:49
  • 2
    $\begingroup$ Look for Googles word2vec which is probably the most successful attempt at that so far. But beware, you need a LOT (as in Google scale) of data for this to be really good. $\endgroup$ – Has QUIT--Anony-Mousse Jan 8 '14 at 18:07
  • $\begingroup$ Yeah, it is for a Hadoop project... We will use a pretty big cluster hehe thanks for your help! What you just said looks exactly like what I was searching! $\endgroup$ – Balduz Jan 8 '14 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.