# Text analysis : What after term-document matrix?

I am trying to build predictive models from text data. I built document-term matrix from the text data (unigram and bigram) and built different types of models on that (like svm, random forest, nearest neighbor etc). All the techniques gave decent results, but I want to improve the results. I tried tuning the models by changing parameters, but that doesn't seem to improve the performance much. What are the possible next steps for me?

• What is your task? Classification? Summarization? Information Retrieval? – Antoine Dec 2 '15 at 8:54

## 1 Answer

Natural language data is usually "noisy" because of the problems like synonymy (different words have the same meaning) and polysemy (the same word has different meaning). You can try to "de-noise" this data by applying dimensionality reduction techniques.

One possibility would be to apply SVD to decompose your document-term matrix as $D = U \Sigma V^T$. If you keep only $k$ largest singular values and approximate $D$ as $D \approx U_k \Sigma_k V_k^T$, what you will get is called "Latent Semantic Analysis": it discovers "latent" concepts in the data set. So you can apply this to your problem and see if it gives a better solution or not.

In scikit learn it would be something like this (code from here):

hasher = HashingVectorizer(n_features=n_features,
stop_words='english', non_negative=True,
norm=None, binary=False)
vectorizer = make_pipeline(hasher, TfidfTransformer())
X = vectorizer.fit_transform(dataset.data)

svd = TruncatedSVD(k)
lsa = make_pipeline(svd, Normalizer(copy=False))

X = lsa.fit_transform(X)

clf = MultinomialNB().fit(X, labels)


Alternatively, you can apply a different decomposition technique called "Non-Negative Matrix Factorization", which also gives an approximate solution $D \approx U V^T$, but all elements of $U$ and $V$ are non-negative.

In scikit learn (code from here):

tfidf = vectorizer.fit_transform(dataset.data)
nmf = NMF(n_components=k, random_state=1)
X = nmf.fit_transform(tfidf)

# do something with X

• note that these techniques are usually applied to unigrams, I'm not sure if they're used for bigrams – Alexey Grigorev May 19 '15 at 13:14
• Tried LSA and got better results. Are there any other NLP techniques which can be applied easily in python? Using nlkt maybe? – Jose May 20 '15 at 20:04
• you can try doing some pre-processing of text data with NLKT (if you're not doing it already), e.g. remove stop words, lemmatize, etc. – Alexey Grigorev May 21 '15 at 6:24