2
$\begingroup$

Forgive me if this is the wrong place for my question. My question is, I hope fairly straightforward. Although I am using Rstudio and the TM package, I think my question is more about the math behind the TFIDF score and is similar to an older question (Typical range of values for TFIDF) but the answer was only partial.

As an example say I have a corpus of 2000 news articles that all contain the word "cat". I used the TFIDF weighting on my Document-Term-Matrix and as expected "cat" receives a low score because it is in every document. However, "dog" appears in 60% of these articles also and it gets a TFIDF score of ~25.0. I would expect that it would have a higher score a long with words like 'tail' (~12.0) etc... but I was under the impression that the scores should be more along the 0-1.0 range rather than something as high as 25!

Basically my returned results match my assumptions, but being that I am new to Text Mining and TFIDF I want to make sure its not a coincidence and that these scores do make sense.

Sorry it is not really replicable, but does a TFIDF score as high as 25 sound plausible?

$\endgroup$
  • $\begingroup$ I once asked something similar : stats.stackexchange.com/questions/165104/… $\endgroup$ – RUser4512 Mar 20 '17 at 14:35
  • $\begingroup$ Hi @RUser4512, thanks, thats why I referenced the question in mine. The problem was there was just an answer to part 2 of your question about dropping low freq words. I am more curious about the 1st part...i just dont quite understand if a TFIDF score of 25 make sense. $\endgroup$ – Yossarianlives Mar 20 '17 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.