7
$\begingroup$

I am working on a text categorization task, and I possess 21,000 documents for training, and (for the time being), 7000 documents for testing. I construct the doc-term matrix for both training corpus and testing corpus, with two different weighting factors, i.e. TF (term frequency) or TF-IDF (term frequency–inverse document frequency). Then I used SVM with Gaussian Radial kernel, to classify the documents. The F1-measure for tf-idf weighting is nearly 0.8, while for tf weighting, the performance is less better, around 0.7. So, logically, We'll be prone to use tf-idf weighting.

However, one problem will come out in an INCREMENTAL CONTEXT. That is, when we have to categorize a single or a few new documents from time to time (with a pre-trained model). It will not be suitable to use tf-idf weighting for one single document, since tf-idf is often used to measure a word importance in a collection of documents.

Should I compromise to use tf weighting or some other tricks exist?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Have you thought about storing the term frequency and the document frequency separately for the trained set, then when you add a new document you can update the document frequency of the new trained set, i.e. including the new document, and calculate tf-idf. Or am I missing something?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes. If your dataset is ~40,000 average sized documents, you'll find the parsing and tokenization is computationally expensive, but the tf-idf step is computationally cheap, and probably much cheaper than the SVM training. So cache the term frequencies (tf) on a document-by-document basis. You can then calculate tf-idf quite quickly, $\endgroup$
    – Patrick Caldon
    Commented Jun 7, 2012 at 1:34
  • $\begingroup$ @PatrickCaldon Yes, both of you are right. By storing a term frequency vector for the whole vocabulary, and when a new document comes, we can easily update the idf vector. By the way, I think it's not necessary to re-train the model with the new idf vector. am I right ? $\endgroup$
    – Ensom Hodder
    Commented Jun 7, 2012 at 5:47
  • $\begingroup$ I think you can wait for a certain percentage of new data before you retrain your set. But never retraining would probably not be wise. $\endgroup$
    – simmmons
    Commented Jun 7, 2012 at 13:31
4
$\begingroup$

The same as simmmons clarified, we can store a term frequency vector for all terms in the training vocabulary. Then we can easily survive in this incremental context, by assuming the new document is also in the training corpus, and then update this vector with newly added files, and finally re-calculate the idf (tf-idf) values for this doc. For simplicity, I think it's not wise to re-train the model for the "out-of-date" training corpus, if the size of training corpus is too large, 1 million for instance.

I found a similar thread as my question, but with more detailed explanations and some maths formulas. You may refer to it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.