A common approach to text classification is to train a classifier off of a 'bag-of-words'. The user takes the text to be classified and counts the frequencies of the words in each object, followed by some sort of trimming to keep the resulting matrix of a manageable size.

Often, I see users construct their feature vector using TFIDF. In other words, the text frequencies noted above are down-weighted by the frequency of the words in corpus. I see why TFIDF would be useful for selecting the 'most distinguishing' words of a given document for, say, display to a human analyst. But in the case of text categorization using standard supervised ML techniques, why bother downweighting by the frequency of documents in the corpus? Will not the learner itself decide the importance to assign to each word/combination of words? I'd be grateful for your thoughts on what value the IDF adds, if any.


The answer is very straight-forward: TF-IDF can achieve better results than simple term frequencies when combined with some supervised methods.

The canonical example is using cosine similarity as a measurement of similarity between documents. Taking the cosine of the angle between the TF-IDF vector representation of documents can successfully retrieve relevant similar documents with higher accuracy than TF alone.

This is because IDF reduces the weight given to common words, and highlights the uncommon words in a document. Most news articles aren't about ostriches, so a news article containing "ostrich" is unusual, and we'd like to know that when trying to find documents that are similar.

But in the case of text categorization using standard supervised ML techniques, why bother downweighting by the frequency of documents in the corpus? Will not the learner itself decide the importance to assign to each word/combination of words?

This illustrates a key point in machine learning: better features tend to beat a cleverer algorithm. An ML tool is just trying to learn a function to map input(s) $x$ to output(s) $y$. If our representation of $x$ is so good that they are already basically $y$ (or, in an ideal case, literally are $y$), then we've made the task much easier on ourselves, and our poor, overworked computers! I think this is an under-appreciated component of the field -- people spend lots of time studying and considering the algorithms because they are domain-independent, but knowing more about your data and the problem you're trying to solve can suggest paths to improved data collection or data representation which make the task so much easier -- and so easy that a model of ornate sophistication is unnecessary.

A number of resources can be found here, which I reproduce for convenience.

  • K. Sparck Jones. "A statistical interpretation of term specificity and its application in retrieval". Journal of Documentation, 28 (1). 1972.

  • G. Salton and Edward Fox and Wu Harry Wu. "Extended Boolean information retrieval". Communications of the ACM, 26 (11). 1983.

  • G. Salton and M. J. McGill. "Introduction to modern information retrieval". 1983

  • G. Salton and C. Buckley. "Term-weighting approaches in automatic text retrieval". Information Processing & Management, 24 (5). 1988.

  • H. Wu and R. Luk and K. Wong and K. Kwok. "Interpreting TF-IDF term weights as making relevance decisions". ACM Transactions on Information Systems, 26 (3). 2008.

  • $\begingroup$ Thanks for the note @user777! Appreciate it. I'm taking a look at those articles. Are there general classes of algorithms that we expect to preferentially benefit from TFIDF vs. just TF? $\endgroup$ – shf8888 May 19 '15 at 22:53
  • $\begingroup$ @shf8888 I'm not sure if there are general classes where one is better. It's possible! As far as I am aware, the first reflex of someone working on an NLP task is to try TF and then TF-IDF as baseline methods before progressing to a more complicated model. This way, you can quantify just how much increased performance you purchase for the increased effort expended by using increasingly complicated models. $\endgroup$ – Sycorax May 19 '15 at 22:57
  • $\begingroup$ Thanks very much! Well, the answer that "empirically TFIDF can provide increased performance over TF with some algorithms" (if you don't object to my one sentence summary) is definitely good from my perspective. Thank you for the references. $\endgroup$ – shf8888 May 19 '15 at 23:03

In the typical case, you could have many more documents in your corpus than labeled documents. That means the IDF can be calculated much more accurately and completely when using the whole corpus.

Next consider the case where the corpus you can get your hands on so far is all labeled or the labeled subset is "big enough". In this case the number of iterations needed for training could possibly be smaller when using TfIDF because the learning algorithm wouldn't need to learn as much.

Finally, in this same case, you could also provide tf only, or tf and idf separately (or even include tfidf as well). I would think this could potentially generate better results, for example, when using a sophisticated kernel function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.