1
$\begingroup$

I want to find the similarity between a document with documents coded as TF-IDF in a pickle file (Python). TF-IDF is done as offline so there is no problem, but when I send a new document for similarity check it takes around 2 minute while I need something real-time (< 2 seconds). For this purpose I used the following code:

for p_tf in p_tfidf:
    temp_similarity = 0
    for item in p_tf:
        (score,word) = item
        if word in input_text:
            temp_similarity += score

    similarity_score.append([temp_similarity, id])

Any clue how to improve system?

$\endgroup$
1
  • $\begingroup$ This question seems to focus on Python coding. Do you think it can be reformulated to highlight some theoretical or practical interest in statistical terms? $\endgroup$
    – chl
    Oct 28, 2013 at 8:22

1 Answer 1

3
$\begingroup$

You can make use of sklearn.feature_extraction.text.TfidfVectorizer

A simple example:

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(min_df=1)

my_phrases = ["boring answer phrase",
              "exciting phrase",
              "phrase on stackoverflow",
              "answer on stackoverflow"]

my_features = vectorizer.fit_transform(my_phrases)

Result:

>>> import numpy as np
>>> np.set_printoptions(precision=4)
>>> my_features.A
array([[ 0.5535,  0.702 ,  0.    ,  0.    ,  0.4481,  0.    ],
       [ 0.    ,  0.    ,  0.8429,  0.    ,  0.538 ,  0.    ],
       [ 0.    ,  0.    ,  0.    ,  0.6137,  0.4968,  0.6137],
       [ 0.5774,  0.    ,  0.    ,  0.5774,  0.    ,  0.5774]])
>>> vectorizer.get_feature_names()
[u'answer', u'boring', u'exciting', u'on', u'phrase', u'stackoverflow']

As a side note, you can remove "stop words" like "on", by passing stop_words='english' parameter:

vectorizer = TfidfVectorizer(min_df=1, stop_words='english')

Edit:

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

# each phrase here could be document in your list 
# of documents
my_phrases = ["boring answer phrase",
              "exciting phrase",
              "phrase on stackoverflow",
              "answer on stackoverflow"]

#  and you want to find the most similar document
#  to this document             
phrase = ["stackoverflow answer"]

# You could do it like this:
vectorizer = TfidfVectorizer(min_df=1, stop_words='english')
all_phrases = phrase + my_phrases
my_features = vectorizer.fit_transform(all_phrases)
scores = (my_features[0, :] * my_features[1:, :].T).A[0]
best_score = np.argmax(scores)
answer = my_phrases[best_score]

Result:

>>> answer
'answer on stackoverflow'
$\endgroup$
3
  • $\begingroup$ Thanks. I haven't worked with sklearn. It seems your code is for when you are trying a new query (the one I exactly want). In another word, when you want to find how similar is your query to existing documents in Corpus. May you also give me the code to analysis all documents and make the whole corpus? $\endgroup$ Oct 28, 2013 at 5:18
  • $\begingroup$ @BigDataLover, I updated my answer. I am not really sure what you mean by the whole corpus, the feature matrix in initial example is the whole corpus (assuming each phrase is a document), but I hope the update is useful. $\endgroup$
    – Akavall
    Oct 29, 2013 at 1:27
  • $\begingroup$ I'm a little late, but suppose you wanted to do this on a corpus with 1,000,000 documents len(my_phrases) == 1000000 and any phrase on the fly, is there a way to do this quickly so that the fit_transform call executes very quickly? $\endgroup$
    – Tim S
    Jul 16, 2015 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.