# Document Similarity Gensim

I am trying to get related documents for a list of 10,000 documents from the same set of 10,000 docs. I am using two algorithms for testing: gensim lsi and gensim similarity. Both give terrible results. How can I improve it?

from gensim import corpora, models, similarities
from nltk.corpus import stopwords
import re

def cleanword(word):
return re.sub(r'\W+', '', word).strip()

def create_corpus(documents):

# remove common words and tokenize
stoplist = stopwords.words('english')
stoplist.append('')
texts = [[cleanword(word) for word in document.lower().split() if cleanword(word) not in stoplist]
for document in documents]

# remove words that appear only once
all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)

texts = [[word for word in text if word not in tokens_once] for text in texts]

dictionary = corpora.Dictionary(texts)
corp = [dictionary.doc2bow(text) for text in texts]

def create_lsi(documents):

corp = create_corpus(documents)
# extract 400 LSI topics; use the default one-pass algorithm
lsi = models.lsimodel.LsiModel(corpus=corp, id2word=dictionary, num_topics=400)
# print the most contributing words (both positively and negatively) for each of the first ten topics
lsi.print_topics(10)

def create_sim_index(documents):
corp = create_corpus(documents)
index = similarities.Similarity('/tmp/tst', corp, num_features=12)
return index

• General comment on your code: You should build your corpus once and then pass your processed corpus into your classification functions. Don't call create_corpus anew each time you want to run a classification. You're unnecessarily repeating work. Mar 9 '14 at 15:30

It's very difficult for us to give you feedback without understanding more about the corpus you are working with or the results you have already received. Here's some general advice anyway.

Parameter Tuning

You set LSI to get the top 400 topics. I think this is way too many topics, especially considering your corpus only has 10K documents. Try reducing this number. Significantly. I'd recommend maybe trying 30-50 topics. Reducing the number of topics should have the added benefit of reducing your processing time as well (depending on how gensim is computing the SVD, anyway).

Speaking of processing time, I mention this above in a comment but I want to restate it: you should only process your corpus once. You don't need to repeat the corpus processing every time you run a classification. Pull create_corpus out of the definition of your classification functions and instead of passing in "documents", pass your processed corpus to these functions.

Function chaining