1
$\begingroup$

I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least important terms every 10.000 documents or so. Since tf-idf is a measure on document level but with the implication of the whole corpus, I'm not sure on how to continuously update it. Thus far, I'm computing it based on this formula:

tf-idf_continuous = (current_tf-idf * (currentNumberOfArticlesContaining_i -1) + tf_ij * log(N)) / currentNumberOfArticlesContaining_i;

with term i, termfrequency of i in document j (tf_ij), N = number of documents in corpus. So, I'm calculating some sort of mean tf-idf. However, I don't think this is a good approach based on the results I get. However, I don't have too much computation power for building the whole index before calculating tf-idf for all instances.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi @Pete. Am I correctly summarizing when I say that you want to construct TFIDF matrix in chunks of 10k, constantly updating the IDF component as you go along? $\endgroup$
    – shf8888
    Commented Aug 24, 2015 at 13:42
  • $\begingroup$ Hum, in chunks of 10k would be fine, however I need to access the information on document level as I'm processing the documents. Thus, I assumed it would be more practical just to update the tfidf be calculating it for every term new (using the mean as updated value). $\endgroup$
    – Pete
    Commented Aug 25, 2015 at 8:30
  • $\begingroup$ Thanks @Pete. I've put a proposed answer below, but not sure if it's getting exactly at your question. Please let me know! $\endgroup$
    – shf8888
    Commented Aug 27, 2015 at 22:34
  • $\begingroup$ Hey, I'm still figuring out a solution :) Thanks, I'll vote when I thought it through $\endgroup$
    – Pete
    Commented Aug 31, 2015 at 19:53
  • $\begingroup$ Thanks @Pete. Take your time! Also, let me know if I haven't understood the question. $\endgroup$
    – shf8888
    Commented Sep 1, 2015 at 21:23

1 Answer 1

Reset to default
1
$\begingroup$

I would propose the following procedure. For each chunk of 10k:

  1. Calculate word frequencies for each text
  2. If the corpus document frequency (df) component does not exist, initialize by using all text word frequencies. Else, update with the counts from the chunk + transforms necessary. You can handle new words by adding in "zero" columns to the old chunks.
  3. Recalculate tfidf for all processed chunks by taking tf and dividing by idf.

Does that work for you? Normal considerations about trimming sparse words, etc. apply.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. It definitely works better than my approach, however I'm not fully happy with the approach, however, I'm not sure whether tf-idf is the measure to go for my problem after all. There is just too many irrelevant words wrt my task. I want to detect emerging trends so I'll have to apply other statistical measures anyway, I guess. Still wondering if there is a general where to apply tf-idf heuristic $\endgroup$
    – Pete
    Commented Sep 21, 2015 at 9:51
  • $\begingroup$ Hi @Pete. Thanks for the note. If you have a measure of success for your task, you could select words that predict / correlate with / are otherwise useful in this task. Even if using a complicated model you can usually extract feature importance and thus select words of interest from the TFIDF matrix. $\endgroup$
    – shf8888
    Commented Dec 30, 2015 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.