1
$\begingroup$

I have a set of documents like:

D1 = "The sky is blue."
D2 = "The sun is bright."
D3 = "The sun in the sky is bright."

and a set of words like:

"sky","land","sea","water","sun","moon"

I want to create a matrix like this:

   x        D1           D2         D3
sky         tf-idf       0          tf-idf
land        0            0          0
sea         0            0          0
water       0            0          0
sun         0            tf-idf     tf-idf
moon        0            0          0

Something like the example table given here: http://www.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html. In the given link, it uses the same words from the document but I need to use the set of words that I have mentioned.

If the particular word is present in the document then I put the tf-idf values, else I put a 0 in the matrix.

Any idea how I might build some sort of matrix like this? Python will be best but R also appreciated.

I am using the following code but am not sure whether I am doing the right thing or not. My code is:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords


train_set = "The sky is blue.", "The sun is bright.", "The sun in the sky is bright." #Documents
test_set = ["sky","land","sea","water","sun","moon"] #Query
stopWords = stopwords.words('english')

vectorizer = CountVectorizer(stop_words = stopWords)
#print vectorizer
transformer = TfidfTransformer()
#print transformer

trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
#print 'Fit Vectorizer to train set', trainVectorizerArray
#print 'Transform Vectorizer to test set', testVectorizerArray

transformer.fit(trainVectorizerArray)
#print
#print transformer.transform(trainVectorizerArray).toarray()

transformer.fit(testVectorizerArray)
#print 
tfidf = transformer.transform(testVectorizerArray)
print tfidf.todense()

I am getting very absurd results like this (values are only 0 and 1 while I am expecting values between 0 and 1).

[[ 0.  0.  1.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  1.]
 [ 0.  0.  0.  0.]
 [ 1.  0.  0.  0.]]   

I am also open to other libraries for calculating tf-idf. I just want a correct matrix which I mentioned above.

$\endgroup$
  • $\begingroup$ Just FYI, you can (at least on sklearn 0.18) tell the vectorizer to avoid stopwords by doing using kwarg stop_words='english' directly, without the need to use NLTK. $\endgroup$ – mar tin Oct 6 '16 at 14:51
4
$\begingroup$

Have a look at gensim or scikit-learn.

Code

from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords


train_set = ["The sky is blue.", "The sun is bright.", "The sun in the sky is bright."]
stop_words = stopwords.words('english')

transformer = TfidfVectorizer(stop_words=stop_words)
transformer.fit_transform(train_set).todense()

After fitting the model, you can transform your out of sample documents.

transformer.transform(test_set).todense()

However, it sounds like what you really want to do given your comments is evaluate the tf-idf of the original documents in terms of the "test_set" as the vocabulary? It's unclear to me what you're after I guess. If that's the case though then something like

transformer = TfidfVectorizer(stop_words=stop_words, vocabulary=test_set)
transformer.fit_transform(train_set).todense().T

Gives you what you want I think.

$\endgroup$
  • $\begingroup$ I have already done that before and I am getting some weird results. See my edited question. $\endgroup$ – user2567857 Jun 2 '14 at 17:12
  • $\begingroup$ Well, you don't actually follow the instructions at your link or mine... See edited answer. $\endgroup$ – jseabold Jun 2 '14 at 18:18
  • 1
    $\begingroup$ Please read my question carefully. By using your method, I am getting an output of 4 columns which corresponds to sky, blue, sun, bright and 3 rows which corresponds to the 3 documents. I need to find the tf-idf values of an external set of words, not the words that appear only in the document. If I use your method, then how can I know whether the external set of words contains words that are already in the document? $\endgroup$ – user2567857 Jun 2 '14 at 18:48
  • 1
    $\begingroup$ Yes, after you fit the model, you use the transform method on your test data. Why do you expect values that aren't 1? In your test set, the documents all contain one word and the default for TfidfVectorizer is to normalize the documents so that the l2 norm of the documents is 1. $\endgroup$ – jseabold Jun 2 '14 at 18:54
  • 1
    $\begingroup$ First you fit the transformer on your training set. Then you can compute the tf-idf on the out of sample documents. It's not clear to me what exactly you want to do but re-reading your question it sounds like you really want to do is use your 'test_set' as a vocabulary. See edits in answer. $\endgroup$ – jseabold Jun 2 '14 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.