I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very rigorously, but can't seem to find anything. Specifically, this is the weighting scheme:

$$ {\rm tfidf}(t,d,D)=(1+\log({f_{t,d})})\cdot \log\!\bigg(1+\frac{N}{n_t}\bigg) \\ N=|D| $$

where $t$ is the query term, $d$ is the document, $D$ is set of documents, $n_t$ is the document frequency of $t$, $f_{t,d}$ is how many times $t$ appears in $d$

  • 1
    $\begingroup$ Please describe what all the notation in the equation means. $\endgroup$ – Greenparker May 15 '16 at 0:01
  • $\begingroup$ What's your question? Is it just a reference request? $\endgroup$ – Sycorax May 15 '16 at 19:40
  • $\begingroup$ @C11H17N2O2SNa Yes. Where did this come from? $\endgroup$ – aces May 15 '16 at 20:05

I was also looking for a reference to justify my use of sublinear tf. I couldn't find where it originated from, but if you just need a reference you can use An introduction to information retrieval, 2009, C.D. Manning et al. Section 6.4 is on variant tf-idf functions that includes sublinear tf scaling.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.