I am trying to build predictive models from text data. I built document-term matrix from the text data (unigram and bigram) and built different types of models on that (like svm, random forest, nearest neighbor etc). All the techniques gave decent results, but I want to improve the results. I tried tuning the models by changing parameters, but that doesn't seem to improve the performance much. What are the possible next steps for me?

  • $\begingroup$ What is your task? Classification? Summarization? Information Retrieval? $\endgroup$
    – Antoine
    Dec 2, 2015 at 8:54

1 Answer 1


Natural language data is usually "noisy" because of the problems like synonymy (different words have the same meaning) and polysemy (the same word has different meaning). You can try to "de-noise" this data by applying dimensionality reduction techniques.

One possibility would be to apply SVD to decompose your document-term matrix as $D = U \Sigma V^T$. If you keep only $k$ largest singular values and approximate $D$ as $D \approx U_k \Sigma_k V_k^T$, what you will get is called "Latent Semantic Analysis": it discovers "latent" concepts in the data set. So you can apply this to your problem and see if it gives a better solution or not.

In scikit learn it would be something like this (code from here):

hasher = HashingVectorizer(n_features=n_features,
                           stop_words='english', non_negative=True,
                           norm=None, binary=False)
vectorizer = make_pipeline(hasher, TfidfTransformer())
X = vectorizer.fit_transform(dataset.data)

svd = TruncatedSVD(k)
lsa = make_pipeline(svd, Normalizer(copy=False))

X = lsa.fit_transform(X)

clf = MultinomialNB().fit(X, labels)

Alternatively, you can apply a different decomposition technique called "Non-Negative Matrix Factorization", which also gives an approximate solution $D \approx U V^T$, but all elements of $U$ and $V$ are non-negative.

In scikit learn (code from here):

tfidf = vectorizer.fit_transform(dataset.data)
nmf = NMF(n_components=k, random_state=1)
X = nmf.fit_transform(tfidf)

# do something with X
  • $\begingroup$ note that these techniques are usually applied to unigrams, I'm not sure if they're used for bigrams $\endgroup$ May 19, 2015 at 13:14
  • $\begingroup$ Tried LSA and got better results. Are there any other NLP techniques which can be applied easily in python? Using nlkt maybe? $\endgroup$
    – Jose
    May 20, 2015 at 20:04
  • $\begingroup$ you can try doing some pre-processing of text data with NLKT (if you're not doing it already), e.g. remove stop words, lemmatize, etc. $\endgroup$ May 21, 2015 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.