12
$\begingroup$

I was reading:

https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition

But I cannot seem to understand exactly why the formula was constructed the way it is.

What I do Understand:

iDF should at some level measure how frequently a term S appears in each of the documents, decreasing in value as the term appears more frequently.

From that perspective

$$ iDF(S) = \frac{\# \text{ of Documents}}{\# \text{ of Documents containing S}}$$

Furthermore term frequency can be rightly described as

$$ tf(S,D) = \frac{\# \ \text{of Occurrences of S in document D}}{\# \ \text{maximum number of occurrences for any string Q in document D}} $$

So then the measure

$$ iDF(S) \times tf(S,D) $$

is in some way proportional to how frequently a term appears in a given document, and how unique that term is over the set of documents.

What I don't understand

But the formula given describes it as

$$ \left( \log(iDF(S)) \right) \left( \frac{1}{2} + \log(\frac{1}{2} tf(S,D)) \right) $$

I wish to understand the need for the logarithms described in the definition. Like, why are they there? What aspect do they emphasize?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
11
$\begingroup$

The aspect emphasised is that the relevance of a term or a document does not increase proportionally with term (or document) frequency. Using a sub-linear function therefore helps dampen down this effect. To that extent, the influence of very large or very small values (e.g. very rare words) is also amortised. Finally, as most people intuitively perceive scoring functions to be somewhat additive, using logarithms will make probability of different independent terms from $P(A, B) = P(A) \, P(B)$ to look more like $\log(P(A,B)) = \log(P(A)) + \log(P(B))$.

As the Wikipedia article you link notes the justification of TF-IDF is still not well-established; it is/was a heuristic that we want to make rigorous, not a rigorous concept we want to transfer to the real world. As mentioned by @Anony-Mousse as a very good read on the matter is Robertson's Understanding Inverse Document Frequency: On theoretical arguments for IDF. It gives a broad overview of the whole framework and attempts to ground TF-IDF methodology to the relevance weighting of search terms.

Improve this answer
$\endgroup$
6
  • 5
    $\begingroup$ Some justification of TF-IDF can be found in "A formal study of information retrieval heuristics.", 2004 by Fang, Hui et al (pdf). $\endgroup$
    – Alexey Grigorev
    Commented Jul 16, 2015 at 7:46
  • 3
    $\begingroup$ I think this is the better reference for TF-IDF justifications: Robertson, S. (2004). "Understanding inverse document frequency: On theoretical arguments for IDF". Journal of Documentation 60 (5): 503–520. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jul 16, 2015 at 8:19
  • $\begingroup$ Thank you for you comments gentlemen (and special thanks Alexey for the correcting the \log, I constantly forget them); +1 to both. I have seen the Robertson paper and contemplated adding it; it is a really good read, I will add it in the main body. $\endgroup$
    – usεr11852
    Commented Jul 16, 2015 at 9:59
  • $\begingroup$ @Anony-Mousse (pdf) $\endgroup$
    – Walrus the Cat
    Commented Dec 13, 2016 at 18:10
  • $\begingroup$ I want to know why "maximum number of occurrences for any string Q in document D" is used instead of number of occurrences for all strings in document D. Why do we want the count of the most common word instead of the count of all words? $\endgroup$
    – Xeoncross
    Commented Oct 12, 2019 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.