# Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common approach I found in most of the papers is that representing each news article as a vector using the vector space model and TF-IDF weights and then cluster those vectors with online clustering algorithm using cosine similarity as a similarity metric. But I have a problem with this approach, specifically using the vector space model.

What is the vocabulary I should use for vector space model in this online environment? If I use a fixed set of terms as my vocabulary, vectors will not be able to represent some rare terms in news articles which not in my vocabulary. However, those rare terms are the most significant features that we can use to group similar articles together. On the other hand, it's not practical to use unlimited or dynamic vocabulary. Can someone suggest me good approach to solve this problem? Or have I misunderstood something? In fact, I am a novice to this research field.

• Did you find any solution to this? – alphacentauri Jun 15 '15 at 22:04

In a streaming environment, we can take the first $n$ documents and calculate the IDF scores and use those to evaluate subsequent documents. Periodically, we can update the IDF scores based on more recent articles. When an article appears with terms that have not previously been seen, we have no estimate of the terms' importance so we simply ignore them. If the term appears in several articles, then next time we recalculate the IDF scores, that term will appear in the list. If a term appears only once (or extremely rarely), then it's probably not a useful feature.