Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to cluster the news that report the same event. Clustering performance is beyond time efficiency.

I think about using a slide window, only clustering news "week per unit"(making an assumption that one event is reported within a week).

Any ideas for this problem? Looking forward for the a suggestions:) Thanks in advance.

  • $\begingroup$ What are you basing your clustering on? What are the variables? Are you looking at what words occur with what frequency within the news articles, or..? $\endgroup$ – Angelorf Mar 20 '14 at 10:39
  • $\begingroup$ Every news article has a time stamp. I want to cluster them based on two variables:(1)time closeness, (2)TF-IDF weights of cosine similarity. My final goal is to cluster those news reporting the same event. But the corpus is large. $\endgroup$ – user3161909 Mar 20 '14 at 11:21
  • $\begingroup$ Making the assumption that each event is reported within one week is of course too coarse. What you could do is split up into two-week batches which overlap with one week. Then you won't disregard or count double the event that have reports carrying over the weekend (falling in two weeks). The problem then is how to combine the results of these separate batches, since they might contain the same / similar clusters. $\endgroup$ – Angelorf Mar 20 '14 at 15:47
  • $\begingroup$ Supposing the clustering algorithm is good and that you find the clusters that are really there, it shouldn't be too hard to figure out which clusters from one two-week batch are 'the same' as clusters from the next. However, things will get difficult when two clusters in one two-week span are regarded as one cluster in the next. $\endgroup$ – Angelorf Mar 20 '14 at 15:50
  • $\begingroup$ On the other hand these things might become more problematic than the problems caused by splitting up all events into non-overlapping batches in the naive way you seemed to suggest. $\endgroup$ – Angelorf Mar 20 '14 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.