I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least important terms every 10.000 documents or so. Since tf-idf is a measure on document level but with the implication of the whole corpus, I'm not sure on how to continuously update it. Thus far, I'm computing it based on this formula:
tf-idf_continuous = (current_tf-idf * (currentNumberOfArticlesContaining_i -1) + tf_ij * log(N)) / currentNumberOfArticlesContaining_i;
with term i, termfrequency of i in document j (tf_ij), N = number of documents in corpus. So, I'm calculating some sort of mean tf-idf. However, I don't think this is a good approach based on the results I get. However, I don't have too much computation power for building the whole index before calculating tf-idf for all instances.