1
$\begingroup$

I am studying tf-idf (term frequency - inverse document frequency). The original logic for tf was straightforward: count of term t / number of total terms in the document.

However, I came across the log scaled frequency: log(1 + count of term t in the document). Please refer to Wikipedia.

It does not include the number of total terms in a document. For example, say, document 1 has 10 words in total and one of them is "happy". Using the original logic, tf(happy)=1/10=0.1. Document 2 also has one "happy" but it has 1,000 words in total. tf(happy)=1/1000=0.001. You can see the tf(happy) of document 1 is very different from that of document 2.

However, if we use the log scaled frequency, both are log(1+1), regardless of the length of documents (one only has 10 words, while the other has 1,000).

How to justify such logic? Thanks.

$\endgroup$

1 Answer 1

-1
$\begingroup$

That means that both documents contain the same instances of the searched word, while having more or less unrelated words doesn't make it more or less relevant than the other document.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.