Term frequency/inverse document frequency (TF/IDF): weighting

I've got a dataset which represents 1000 documents and all the words that appear in it. So the rows represent the documents and the columns represent the words. So for example, the value in cell $(i,j)$ stands for the times word $j$ occurs in document $i$. Now, I have to find 'weights' of the words, using tf/idf method, but I actually don't know how to do this. Can someone please help me out?

Wikipedia has a good article on the topic, complete with formulas. The values in your matrix are the term frequencies. You just need to find the idf: (log((total documents)/(number of docs with the term)) and multiple the 2 values.

In R, you could do so as follows:

set.seed(42)
d <- data.frame(w=sample(LETTERS, 50, replace=TRUE))
d <- model.matrix(~0+w, data=d)

tf <- d
idf <- log(nrow(d)/colSums(d))
tfidf <- d

for(word in names(idf)){
tfidf[,word] <- tf[,word] * idf[word]
}


Here's the datasets:

> colSums(d)
wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ
3  1  3  1  1  1  1  2  4  2  2  1  1  3  2  2  2  4  5  5  4
wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ
1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0
2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0
3  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
4  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0
5  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0
6  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0
wA wC wD wF wG   wH wJ wK wL wM   wN wO wP   wQ wR wS wT   wV  wX  wY wZ
1  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 2.3 0.0  0
2  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 2.3  0
3  0  0  0  0  0 3.91  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 0.0  0
4  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 2.53 0.0 0.0  0
5  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 2.81  0  0  0 0.00 0.0 0.0  0
6  0  0  0  0  0 0.00  0  0  0  0 3.22  0  0 0.00  0  0  0 0.00 0.0 0.0  0


You can also look at the idf of each term:

> log(nrow(d)/colSums(d))
wA       wC       wD       wF       wG       wH       wJ       wK       wL       wM       wN       wO       wP       wQ       wR       wS       wT       wV       wX       wY       wZ
2.813411 3.912023 2.813411 3.912023 3.912023 3.912023 3.912023 3.218876 2.525729 3.218876 3.218876 3.912023 3.912023 2.813411 3.218876 3.218876 3.218876 2.525729 2.302585 2.302585 2.525729

• Thanks for your help! But is it possible to obtain some value for each word which represents some weighting (instead of a whole matrix)? Now we have a whole matrix of weights. I'm doing some feature selection and want to use tf/idf as a filter method... – ABC Dec 2 '13 at 17:25
• @ABC tf-idf by definition refers to the full matrix of weights. Perhaps you are interested in the idf weights alone, which you would get by log((number of docs)/(number of docs containing the term)). You could also just filter out the infrequent terms. – Zach Dec 2 '13 at 18:50
• Very clear! Really appreciated. – ABC Dec 2 '13 at 19:00

there is package tm (text mining) http://cran.r-project.org/web/packages/tm/index.html which should do exactly you need:

#read 1000 txt articles from directory data/txt
#some preprocessing
corpus <- tm_map(corpus, removeWords, stopwords("english"))
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, stemDocument, language="english")
#creating term matrix with TF-IDF weighting
terms <-DocumentTermMatrix(corpus,control = list(weighting = function(x) weightTfIdf(x, normalize = FALSE)))

#or compute cosine distance among documents
dissimilarity(tdm, method = "cosine")


R is a functional language so reading code can be tricky (e.g. x in terms)

Your code has an error: colSums computes the number of occurence in the corpus, not the number of texts with the word.

A version computing such would be:

tfidf=function(mat){
tf <- mat
id=function(col){sum(!col==0)}
idf <- log(nrow(mat)/apply(mat, 2, id))
tfidf <- mat
for(word in names(idf)){tfidf[,word] <- tf[,word] * idf[word]}
return(tfidf)
}


There is a new R package which can do this: textir: Inverse Regression for Text Analysis

The relevant command is tfidf, the example from the manual:

data(we8there)
## 20 high-variance tf-idf terms
colnames(we8thereCounts)[
order(-sdev(tfidf(we8thereCounts)))[1:20]]


I am late to this party, but I was playing with the concepts of tc-idf (I want to emphasize the word 'concept' because I didn't follow any books for the actual calculations; so they may be somewhat off, and definitely more easily carried out with packages such as {tm: Text Mining Package}, as mentioned), and I think what I got may be related to this question, or, in any event, this may be a good place to post it.

SET-UP: I have a corpus of 5 long paragraphs taken from printed media, text 1 through 5 such as The New York Times. Allegedly, it is a very small "body", a tiny library, so to speak, but the entries in this "digital" library are not random: The first and fifth entries deal with football (or 'soccer' for 'social club' (?) around here), and more specifically about the greatest team today. So, for instance, text 1 begins as...

"Over the past nine years, Messi has led F.C. Barcelona to national and international titles while breaking individual records in ways that seem otherworldly..."

Very nice! On the other hand you would definitely want to skip the contents in the three entries in between. Here's an example (text 2):

"In the span of a few hours across Texas, Mr. Rubio suggested that Mr. Trump had urinated in his trousers and used illegal immigrants to tap out his unceasing Twitter messages..."

So what to do to avoid at all cost "surfing" from the text 1 to text 2, while continuing to rejoice in the literature about almighty Barcelona F.C. in text 5?

TC-IDF: I isolated the words in every text into long vectors. Then counted the frequency of each word, creating five vectors (one for each text) in which only the words encountered in the corresponding text were counted - all the other words, belonging to other texts, were valued at zero. In the first snippet of text 1, for instance, its vector would have a count of 1 for the word "Messi", while "Trump" would have 0. This was the tc part.

The idf part was also calculated separately for each text, and resulted in 5 "vectors" (I think I treated them as data frames), containing the logarithmic transformations of the counts of documents (sadly, just from zero to five, given our small library) containing a given word as in:

$\log\left(\frac{\text{No. documents}}{1\, +\, \text{No. docs containing a word}}\right)$. The number of documents is 5. Here comes the part that may answer the OP: for each idf calculation, the text under consideration was excluded from the tally. But if a word appeared in all documents, its idf was still $0$ thanks to the $1$ in the denominator - e.g. the word "the" had importance 0, because it was present in all texts.

The entry-wise multiplication of $\text{tc} \times \text{idf}$ for every text was the importance of every word for each one of the library items - locally prevalent, globally rare words.

COMPARISONS: Now it was just a matter of performing dot products among these "vectors of word importance".

Predictably, the dot product of text 1 with text 5 was 13.42645, while text 1 v. text2 was only 2.511799.

The clunky R code (nothing to imitate) is here.

Again, this is a very rudimentary simulation, but I think it is very graphic.