From what I understand, the "term frequency" ("TF" in TF-IDF) is calculated from the number of times a certain term occurs in a document vs the most common term in a document:

0.5 * (1 + float64(countOfThisTermInThisDoc) / float64(countOfMostCommonTermInThisDoc)) 

Why is the number of times the most common word (in the document) used instead of the total number of words?

0.5 * (1 + float64(countOfThisTermInThisDoc) / float64(countOfAllTermsInDoc)) 
  • $\begingroup$ @roundsquare I think you're mixing the TermFrequency (TF) with the Inverse Document Frequency (IDF). TF is only for a single document. $\endgroup$
    – Xeoncross
    Oct 12 '19 at 16:15
  • $\begingroup$ You are correct! I misread your question. I've deleted my comment - please see Tim's answer, its much better. $\endgroup$ Oct 12 '19 at 17:22

It does not. Term frequency is the number of times that you observed the term in the document. Inverse document frequency is logarithm of the total number of documents divided by number of documents that contain the term at least once. To get TF-IDF, you just multiply them.

That said, as already noticed in the comments, there are modifications of TF-IDF. Dividing by things like total numbers of words in document, count of most frequent term, number of words in document, etc., are all used to correct for things like documents of different lengths, skewed distributions of counts, etc. People choose among those modifications either based on their knowledge about the data (e.g. documents significantly vary in length), or empirically (try few, choose the one that gives best results).

  • $\begingroup$ It appears my comment, that you are referring to, was based on a misreading of the question. Nevertheless, what Tim says is correct - the different formulations you gave are ways to modify TF in order to correct for various traits of your dataset (e.g. documents of highly variable length). $\endgroup$ Oct 12 '19 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.