# How to continuously compute tf-idf for relevance of single terms

I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least important terms every 10.000 documents or so. Since tf-idf is a measure on document level but with the implication of the whole corpus, I'm not sure on how to continuously update it. Thus far, I'm computing it based on this formula:

tf-idf_continuous = (current_tf-idf * (currentNumberOfArticlesContaining_i -1) + tf_ij * log(N)) / currentNumberOfArticlesContaining_i;

with term i, termfrequency of i in document j (tf_ij), N = number of documents in corpus. So, I'm calculating some sort of mean tf-idf. However, I don't think this is a good approach based on the results I get. However, I don't have too much computation power for building the whole index before calculating tf-idf for all instances.

• Hi @Pete. Am I correctly summarizing when I say that you want to construct TFIDF matrix in chunks of 10k, constantly updating the IDF component as you go along? Commented Aug 24, 2015 at 13:42
• Hum, in chunks of 10k would be fine, however I need to access the information on document level as I'm processing the documents. Thus, I assumed it would be more practical just to update the tfidf be calculating it for every term new (using the mean as updated value).
– Pete
Commented Aug 25, 2015 at 8:30
• Thanks @Pete. I've put a proposed answer below, but not sure if it's getting exactly at your question. Please let me know! Commented Aug 27, 2015 at 22:34
• Hey, I'm still figuring out a solution :) Thanks, I'll vote when I thought it through
– Pete
Commented Aug 31, 2015 at 19:53
• Thanks @Pete. Take your time! Also, let me know if I haven't understood the question. Commented Sep 1, 2015 at 21:23