I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least important terms every 10.000 documents or so. Since tf-idf is a measure on document level but with the implication of the whole corpus, I'm not sure on how to continuously update it. Thus far, I'm computing it based on this formula:

tf-idf_continuous = (current_tf-idf * (currentNumberOfArticlesContaining_i -1) + tf_ij * log(N)) / currentNumberOfArticlesContaining_i;

with term i, termfrequency of i in document j (tf_ij), N = number of documents in corpus. So, I'm calculating some sort of mean tf-idf. However, I don't think this is a good approach based on the results I get. However, I don't have too much computation power for building the whole index before calculating tf-idf for all instances.

  • $\begingroup$ Hi @Pete. Am I correctly summarizing when I say that you want to construct TFIDF matrix in chunks of 10k, constantly updating the IDF component as you go along? $\endgroup$ – shf8888 Aug 24 '15 at 13:42
  • $\begingroup$ Hum, in chunks of 10k would be fine, however I need to access the information on document level as I'm processing the documents. Thus, I assumed it would be more practical just to update the tfidf be calculating it for every term new (using the mean as updated value). $\endgroup$ – Pete Aug 25 '15 at 8:30
  • $\begingroup$ Thanks @Pete. I've put a proposed answer below, but not sure if it's getting exactly at your question. Please let me know! $\endgroup$ – shf8888 Aug 27 '15 at 22:34
  • $\begingroup$ Hey, I'm still figuring out a solution :) Thanks, I'll vote when I thought it through $\endgroup$ – Pete Aug 31 '15 at 19:53
  • $\begingroup$ Thanks @Pete. Take your time! Also, let me know if I haven't understood the question. $\endgroup$ – shf8888 Sep 1 '15 at 21:23

I would propose the following procedure. For each chunk of 10k:

  1. Calculate word frequencies for each text
  2. If the corpus document frequency (df) component does not exist, initialize by using all text word frequencies. Else, update with the counts from the chunk + transforms necessary. You can handle new words by adding in "zero" columns to the old chunks.
  3. Recalculate tfidf for all processed chunks by taking tf and dividing by idf.

Does that work for you? Normal considerations about trimming sparse words, etc. apply.

  • $\begingroup$ Thanks for the answer. It definitely works better than my approach, however I'm not fully happy with the approach, however, I'm not sure whether tf-idf is the measure to go for my problem after all. There is just too many irrelevant words wrt my task. I want to detect emerging trends so I'll have to apply other statistical measures anyway, I guess. Still wondering if there is a general where to apply tf-idf heuristic $\endgroup$ – Pete Sep 21 '15 at 9:51
  • $\begingroup$ Hi @Pete. Thanks for the note. If you have a measure of success for your task, you could select words that predict / correlate with / are otherwise useful in this task. Even if using a complicated model you can usually extract feature importance and thus select words of interest from the TFIDF matrix. $\endgroup$ – shf8888 Dec 30 '15 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.