A common approach to text classification is to train a classifier off of a 'bag-of-words'. The user takes the text to be classified and counts the frequencies of the words in each object, followed by some sort of trimming to keep the resulting matrix of a manageable size.

Often, I see users construct their feature vector using TFIDF. In other words, the text frequencies noted above are down-weighted by the frequency of the words in corpus. I see why TFIDF would be useful for selecting the 'most distinguishing' words of a given document for, say, display to a human analyst. But in the case of text categorization using standard supervised ML techniques, why bother downweighting by the frequency of documents in the corpus? Will not the learner itself decide the importance to assign to each word/combination of words? I'd be grateful for your thoughts on what value the IDF adds, if any.


You're correct that the supervised learner can often be redundant with TF-IDF weighting. Here's the basic outline of why: In one typical form of TF-IDF weighting, the rescaling is logarithmic, so the weighting for a word $w$ in a document $d$ is $$ \text{TF-IDF}(w,d) = (\text{no. occurrences of $w$ in $d$}) \cdot f(w) $$ for $N$ the number of documents in the corpus and $f(w)=\log\left(\frac{N}{\text{no. documents containing $w$}}\right)$. When $f(w)>0$, TF-IDF just amounts to a rescaling of the term frequency. So if we write the matrix counting the number of occurrences of a word in each document as $X$, then a linear model has the form $X\beta$. If we use TF-IDF instead of just term frequency alone, the linear model can be written as $X(k I)\tilde{\beta}$, where $k$ is a vector storing all of our weights $k_i=f(w_i)$. The effect of $kI$ is to rescale each column of $X$. In this setting, the choice to use TF-IDF or TF alone is inconsequential, because you'll get the same predictions. Using the substitution $(kI)\tilde{\beta}=\beta$, we can see the effect is to rescale $\beta$.

But there are at least two scenarios where the choice to use TF-IDF is consequential for supervised learning.

The first case is when $f(w)=0$. This happens whenever a term occurs in every document, such as very common words like "and" or "the." In this case, TF-IDF will zero out the column in $X(kI)$, resulting in a matrix which is not full-rank. A rank-deficient matrix is often not preferred for supervised learning, so instead these words are simply dropped from $X$ because they add no information. In this way, TF-IDF provides automatic screening for the most common words.

The second case is when the matrix $X(kI)$ has its document vectors rescaled to the same norm. Since a longer document is very likely to have a much larger vocabulary than a shorter document, it can be hard to compare documents of different lengths. Rescaling each document vector will also suppress importance rare words in the document independently of how rare or common the word is in the corpus. Moreover, rescaling each document's vector to have the same norm after computing TF-IDF gives a design matrix which is not a linear transformation of $X$, so original matrix cannot be recovered using a linear scaling.

Rescaling the document vectors has a close connection to cosine similarity, since both methods involve comparing unit-length vectors.

The popularity of TF-IDF in some settings does not necessarily impose a limitation on the methods you use. Recently, it has become very common to use word and token vectors that are either pre-trained on a large corpus or trained by the researcher for their particular task. Depending on what you're doing and scale of the data, and the goal of your analysis, it might be more expedient to use TD-IDF, word2vec, or another method to represent natural language information.

A number of resources can be found here, which I reproduce for convenience.

  • K. Sparck Jones. "A statistical interpretation of term specificity and its application in retrieval". Journal of Documentation, 28 (1). 1972.

  • G. Salton and Edward Fox and Wu Harry Wu. "Extended Boolean information retrieval". Communications of the ACM, 26 (11). 1983.

  • G. Salton and M. J. McGill. "Introduction to modern information retrieval". 1983

  • G. Salton and C. Buckley. "Term-weighting approaches in automatic text retrieval". Information Processing & Management, 24 (5). 1988.

  • H. Wu and R. Luk and K. Wong and K. Kwok. "Interpreting TF-IDF term weights as making relevance decisions". ACM Transactions on Information Systems, 26 (3). 2008.

  • $\begingroup$ Thanks for the note @user777! Appreciate it. I'm taking a look at those articles. Are there general classes of algorithms that we expect to preferentially benefit from TFIDF vs. just TF? $\endgroup$ – shf8888 May 19 '15 at 22:53
  • $\begingroup$ @shf8888 I'm not sure if there are general classes where one is better. It's possible! As far as I am aware, the first reflex of someone working on an NLP task is to try TF and then TF-IDF as baseline methods before progressing to a more complicated model. This way, you can quantify just how much increased performance you purchase for the increased effort expended by using increasingly complicated models. $\endgroup$ – Sycorax May 19 '15 at 22:57
  • $\begingroup$ Thanks very much! Well, the answer that "empirically TFIDF can provide increased performance over TF with some algorithms" (if you don't object to my one sentence summary) is definitely good from my perspective. Thank you for the references. $\endgroup$ – shf8888 May 19 '15 at 23:03
  • $\begingroup$ The question asks about classification tasks, but this answer mentions only unsupervised methods (even though it calls them supervised) $\endgroup$ – dmh Mar 16 at 21:57
  • $\begingroup$ @MONODA43 Because both supervised and unsupervised methods use feature engineering (such as TF-IDF or another method), and because this answer devotes a paragraph to supervised learning, characterized as "[a]n ML tool is just trying to learn a function to map input(s) $x$ to output(s) $y$," it is challenging to discern any part of your comment that is true. $\endgroup$ – Sycorax Mar 16 at 22:13

In the typical case, you could have many more documents in your corpus than labeled documents. That means the IDF can be calculated much more accurately and completely when using the whole corpus.

Next consider the case where the corpus you can get your hands on so far is all labeled or the labeled subset is "big enough". In this case the number of iterations needed for training could possibly be smaller when using TfIDF because the learning algorithm wouldn't need to learn as much.

Finally, in this same case, you could also provide tf only, or tf and idf separately (or even include tfidf as well). I would think this could potentially generate better results, for example, when using a sophisticated kernel function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.