Incorporating new words in tfidf feature-vector for online clustering

Question

I am building an Online news clustering system using Lucene and Mahout libraries in java. I intend to use vector space model and tfidf weights for Kmeans(or fuzzy/streamKmeans). My plan is : Cluster initial articles,assign new article to the cluster whose centroid is closest based on a small distance threshold. The leftover documents that aren’t associated with any old clusters form new data(new topics). Separately cluster them among themselves and add these temporary cluster centroids to the previous centroids. Less frequently, execute the full batch clustering to recluster the entire set of documents. The problem arises in comparing a new article to a centroid to assign it to an old cluster. The centroid dimension is number of distinct words in initial data. But the dimension of new article is different. I am following the book Mahout in Action. Is there any approach or some sort of feature extraction to handle this. The following similar links still remain unanswered: bag of words in an online configuration, for classification / clustering Vector Space Model for Online News Clustering Thanks in advance

Answer 1

As I understand it, your problem is that there are new words in the new document that have not been seen in any previous documents. As a result, if you make a tfidf matrix using the words in all the previous documents + the new document, you ends up with a matrix of higher dimension than you previously had for the other documents.

I'd suggest updating the tfidf matrices for the previous documents using the new words. The new columns will be all zero for the previous documents, as they don't have the new words. You can just keep doing this in an online manner, and then you can measure distances using these revised matrices. Definitely use sparse matrices, as this'll get big.