0
$\begingroup$

I am working on a problem of finding similar documents. I am using a Tf-Idf based vector space model representation of documents and it gives me good results.

However when I encounter a document with new words which is out of the current vector space, it gives an error. For example I have two documents represented as vectors of terms with their weights Document A -> (A,B,C)--> ( 0.15,0.4,0.3) Document B->(C,D,E)--> (0.12,0.3,0.5)

How to compute similarity in these situations and so on

Is there an implementation in python or nltk which takes care of incrementally extending the vector space as new documents are entered.

$\endgroup$
2
$\begingroup$

You will almost always find new words in a new document, maybe because there are indeed new words or because typos in the text. I think your best choice is to pick in your word space the most frequent or significant words in your corpora, and then categorize all the non frequent words with a token (i.e "NON_FREQUENT"). Then in you new text you read do the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.