10
$\begingroup$

I was reading:

https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition

But I cannot seem to understand exactly why the formula was constructed the way it is.

What I do Understand:

iDF should at some level measure how frequently a term S appears in each of the documents, decreasing in value as the term appears more frequently.

From that perspective

$$ iDF(S) = \frac{\# \text{ of Documents}}{\# \text{ of Documents containing S}}$$

Furthermore term frequency can be rightly described as

$$ tf(S,D) = \frac{\# \ \text{of Occurrences of S in document D}}{\# \ \text{maximum number of occurrences for any string Q in document D}} $$

So then the measure

$$ iDF(S) \times tf(S,D) $$

is in some way proportional to how frequently a term appears in a given document, and how unique that term is over the set of documents.

What I don't understand

But the formula given describes it as

$$ \left( \log(iDF(S)) \right) \left( \frac{1}{2} + \log(\frac{1}{2} tf(S,D)) \right) $$

I wish to understand the need for the logarithms described in the definition. Like, why are they there? What aspect do they emphasize?

$\endgroup$
8
$\begingroup$

The aspect emphasised is that the relevance of a term or a document does not increase proportionally with term (or document) frequency. Using a sub-linear function therefore helps dumped down this effect. To that extend the influence of very large or very small values (e.g. very rare words) is also amortised. Finally as most people intuitively perceive scoring functions to be somewhat additive using logarithms will make probability of different independent terms from $P(A, B) = P(A) \, P(B)$ to look more like $\log(P(A,B)) = \log(P(A)) + \log(P(B))$.

As the Wikipedia article you link notes the justification of TF-IDF is still not well-established; it is/was a heuristic that we want to make rigorous, not a rigorous concept we want to transfer to the real world. As mentioned by @Anony-Mousse as a very good read on the matter is Robertson's Understanding Inverse Document Frequency: On theoretical arguments for IDF. It gives a broad overview of the whole framework and attempts to ground TF-IDF methodology to the relevance weighting of search terms.

$\endgroup$
  • 4
    $\begingroup$ Some justification of TF-IDF can be found in "A formal study of information retrieval heuristics.", 2004 by Fang, Hui et al (pdf). $\endgroup$ – Alexey Grigorev Jul 16 '15 at 7:46
  • 3
    $\begingroup$ I think this is the better reference for TF-IDF justifications: Robertson, S. (2004). "Understanding inverse document frequency: On theoretical arguments for IDF". Journal of Documentation 60 (5): 503–520. $\endgroup$ – Anony-Mousse Jul 16 '15 at 8:19
  • $\begingroup$ Thank you for you comments gentlemen (and special thanks Alexey for the correcting the \log, I constantly forget them); +1 to both. I have seen the Robertson paper and contemplated adding it; it is a really good read, I will add it in the main body. $\endgroup$ – usεr11852 Jul 16 '15 at 9:59
  • $\begingroup$ @Anony-Mousse (pdf) $\endgroup$ – Walrus the Cat Dec 13 '16 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.