I am trying to get related documents for a list of 10,000 documents from the same set of 10,000 docs. I am using two algorithms for testing: gensim lsi and gensim similarity. Both give terrible results. How can I improve it?

from gensim import corpora, models, similarities
from nltk.corpus import stopwords
import re

def cleanword(word):
    return re.sub(r'\W+', '', word).strip()

def create_corpus(documents):

    # remove common words and tokenize
    stoplist = stopwords.words('english')
    stoplist.append('')
    texts = [[cleanword(word) for word in document.lower().split() if cleanword(word) not in stoplist]
             for document in documents]

    # remove words that appear only once
    all_tokens = sum(texts, [])
    tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)

    texts = [[word for word in text if word not in tokens_once] for text in texts]

    dictionary = corpora.Dictionary(texts)
    corp = [dictionary.doc2bow(text) for text in texts]

def create_lsi(documents):

    corp = create_corpus(documents)
    # extract 400 LSI topics; use the default one-pass algorithm
    lsi = models.lsimodel.LsiModel(corpus=corp, id2word=dictionary, num_topics=400)
    # print the most contributing words (both positively and negatively) for each of the first ten topics
    lsi.print_topics(10)

def create_sim_index(documents):
    corp = create_corpus(documents)
    index = similarities.Similarity('/tmp/tst', corp, num_features=12)
    return index
  • General comment on your code: You should build your corpus once and then pass your processed corpus into your classification functions. Don't call create_corpus anew each time you want to run a classification. You're unnecessarily repeating work. – David Marx Mar 9 '14 at 15:30
up vote 2 down vote accepted

It's very difficult for us to give you feedback without understanding more about the corpus you are working with or the results you have already received. Here's some general advice anyway.

Parameter Tuning

You set LSI to get the top 400 topics. I think this is way too many topics, especially considering your corpus only has 10K documents. Try reducing this number. Significantly. I'd recommend maybe trying 30-50 topics. Reducing the number of topics should have the added benefit of reducing your processing time as well (depending on how gensim is computing the SVD, anyway).

Speaking of processing time, I mention this above in a comment but I want to restate it: you should only process your corpus once. You don't need to repeat the corpus processing every time you run a classification. Pull create_corpus out of the definition of your classification functions and instead of passing in "documents", pass your processed corpus to these functions.

Function chaining

Your question is somewhat confusing:

I am trying to get related documents for a list of 10,000 documents from the same set of 10,000 docs. I am using two algorithms for testing: gensim lsi and gensim similarity. Both give terrible results.

The output of LSI as you are using it is not a list of documents, it's "topics" as defined by combining terms (or rather, if the gensim LSI method gives you documents, it's just giving you back your corpus projected onto a different basis). If you want to use LSI to get related documents, you should be applying your similarity measurement (cosine similarity) in LSI space. The gensim tutorial even suggests this method.

So in short:

  1. Process your corpus only once.

  2. Pass the output to LSI to reduce your document representations from term space to topic space (probably using fewer topics than you are now)

  3. Pass the result of LSI to your similarity function to perform distance measurements in topic space instead of raw term space.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.