# Choosing the best set of keywords

I have a dataset of tweets collected using twitter streaming API on a particular topic (say 'football') using around 40 keywords. Now if I'm going to track the same topic (football) in future how do I determine the best set of keywords to query. Any researchers working on these areas? How do I solve this keyword ambiguity? What are the sophisticated and state of the art techniques?

In terms of a general response, there are definitely some excellent resources out there and lots of research going on in the space. I will focus first on topic models, since I used that in my previous work position and have done some research in this area.

As a starting point, you might want to read an excellent paper on probabilistic topic models by Dr. David Blei. As Dr. Blei notes in the article:

"Topic models are algorithms for discovering the main themes that pervade a large and otherwise unstructured collection of documents."


In the paper, he surveys a number of different approaches that can be used. This provides tremendous perspective.

If following this, you are more interested in getting hands on with the specific technique of topic modelling, and are comfortable with say something like R, you could have a look at the following vignette which looks at topic models in R. It provides a good mix of both theory and R code.

If you want to go even farther, you could also read the package notes for topicmodels: An R Package for Fitting Topic Models. This work, as well as the others, should serve as a great starting point, while also reinforcing the fact that research and resources in this area are abundant (with lots of room for growth and new knowledge).

• Thanks for your ans and the paper, I'll have a look. How about using an query expansion approach? First I'll start with say 5 keywords (out of 40) and then I'll apply a ranking algorithm (IG or Jaccard or Dice ..) and examine the ranked tokens (s1).This s1 may contain keywords as well as non keywords (stopwords are removed). Now I could take some top ranked tokens and add them to the initial 5 keywords and rerank again. And to carry on this process untill maybe I get the same set of tokens at the top (convergence). What do think of something like this? – samsamara Apr 16 '14 at 0:59
• That sounds interesting ... a very interesting approach. Let me look into it tonight and then I can comment back some thoughts or share some resource links. – Nathaniel Payne Apr 16 '14 at 4:17
• Hi @Nathaniel did you happen to think it over? – samsamara Apr 19 '14 at 5:42
• Sorry there @user601! I am stuck working on a project with a hard deadline for Sunday at around 8pm PST. Will have a look at this once I get clear. – Nathaniel Payne Apr 19 '14 at 6:34