I have a set of documents like:

D1 = "The sky is blue."
D2 = "The sun is bright."
D3 = "The sun in the sky is bright."

and a set of words like:


I want to create a matrix like this:

   x        D1           D2         D3
sky         tf-idf       0          tf-idf
land        0            0          0
sea         0            0          0
water       0            0          0
sun         0            tf-idf     tf-idf
moon        0            0          0

Something like the example table given here: http://www.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html. In the given link, it uses the same words from the document but I need to use the set of words that I have mentioned.

If the particular word is present in the document then I put the tf-idf values, else I put a 0 in the matrix.

Any idea how I might build some sort of matrix like this? Python will be best but R also appreciated.

I am using the following code but am not sure whether I am doing the right thing or not. My code is:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords

train_set = "The sky is blue.", "The sun is bright.", "The sun in the sky is bright." #Documents
test_set = ["sky","land","sea","water","sun","moon"] #Query
stopWords = stopwords.words('english')

vectorizer = CountVectorizer(stop_words = stopWords)
#print vectorizer
transformer = TfidfTransformer()
#print transformer

trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
#print 'Fit Vectorizer to train set', trainVectorizerArray
#print 'Transform Vectorizer to test set', testVectorizerArray

#print transformer.transform(trainVectorizerArray).toarray()

tfidf = transformer.transform(testVectorizerArray)
print tfidf.todense()

I am getting very absurd results like this (values are only 0 and 1 while I am expecting values between 0 and 1).

[[ 0.  0.  1.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  1.]
 [ 0.  0.  0.  0.]
 [ 1.  0.  0.  0.]]   

I am also open to other libraries for calculating tf-idf. I just want a correct matrix which I mentioned above.

  • $\begingroup$ Just FYI, you can (at least on sklearn 0.18) tell the vectorizer to avoid stopwords by doing using kwarg stop_words='english' directly, without the need to use NLTK. $\endgroup$ Oct 6, 2016 at 14:51

1 Answer 1


Have a look at gensim or scikit-learn.


from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords

train_set = ["The sky is blue.", "The sun is bright.", "The sun in the sky is bright."]
stop_words = stopwords.words('english')

transformer = TfidfVectorizer(stop_words=stop_words)

After fitting the model, you can transform your out of sample documents.


However, it sounds like what you really want to do given your comments is evaluate the tf-idf of the original documents in terms of the "test_set" as the vocabulary? It's unclear to me what you're after I guess. If that's the case though then something like

transformer = TfidfVectorizer(stop_words=stop_words, vocabulary=test_set)

Gives you what you want I think.

  • $\begingroup$ I have already done that before and I am getting some weird results. See my edited question. $\endgroup$ Jun 2, 2014 at 17:12
  • $\begingroup$ Well, you don't actually follow the instructions at your link or mine... See edited answer. $\endgroup$
    – jseabold
    Jun 2, 2014 at 18:18
  • 1
    $\begingroup$ Please read my question carefully. By using your method, I am getting an output of 4 columns which corresponds to sky, blue, sun, bright and 3 rows which corresponds to the 3 documents. I need to find the tf-idf values of an external set of words, not the words that appear only in the document. If I use your method, then how can I know whether the external set of words contains words that are already in the document? $\endgroup$ Jun 2, 2014 at 18:48
  • 1
    $\begingroup$ Yes, after you fit the model, you use the transform method on your test data. Why do you expect values that aren't 1? In your test set, the documents all contain one word and the default for TfidfVectorizer is to normalize the documents so that the l2 norm of the documents is 1. $\endgroup$
    – jseabold
    Jun 2, 2014 at 18:54
  • 1
    $\begingroup$ First you fit the transformer on your training set. Then you can compute the tf-idf on the out of sample documents. It's not clear to me what exactly you want to do but re-reading your question it sounds like you really want to do is use your 'test_set' as a vocabulary. See edits in answer. $\endgroup$
    – jseabold
    Jun 2, 2014 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.