3
$\begingroup$

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common approach I found in most of the papers is that representing each news article as a vector using the vector space model and TF-IDF weights and then cluster those vectors with online clustering algorithm using cosine similarity as a similarity metric. But I have a problem with this approach, specifically using the vector space model.

What is the vocabulary I should use for vector space model in this online environment? If I use a fixed set of terms as my vocabulary, vectors will not be able to represent some rare terms in news articles which not in my vocabulary. However, those rare terms are the most significant features that we can use to group similar articles together. On the other hand, it's not practical to use unlimited or dynamic vocabulary. Can someone suggest me good approach to solve this problem? Or have I misunderstood something? In fact, I am a novice to this research field.

Improve this question
$\endgroup$
1
  • $\begingroup$ Did you find any solution to this? $\endgroup$
    – alphacentauri
    Commented Jun 15, 2015 at 22:04

2 Answers 2

Reset to default
1
$\begingroup$

A practical solution is to regard the IDF scores as approximate measures of the importance of terms. The more documents we use to estimate the scores, the better the approximation, but it will never be "perfect".

In a streaming environment, we can take the first $n$ documents and calculate the IDF scores and use those to evaluate subsequent documents. Periodically, we can update the IDF scores based on more recent articles. When an article appears with terms that have not previously been seen, we have no estimate of the terms' importance so we simply ignore them. If the term appears in several articles, then next time we recalculate the IDF scores, that term will appear in the list. If a term appears only once (or extremely rarely), then it's probably not a useful feature.

Improve this answer
$\endgroup$
0
$\begingroup$

One approach is to build the vocabulary from the articles. Calculate the TFIDF for all the terms occurring in the articles except the words which occur too frequently (often called stop words, you can get ready made scripts for stop word removal. This is a recent paper on document clustering .

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the answer... The problem is we don't have the articles beforehand as this happen in online environment... we can find set of articles to initialize the system, but it has to work for new articles as well $\endgroup$
    – Tharaka Wijebandara
    Commented Nov 14, 2014 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.