Support vector machine for text classification

Question

I am currently having a data set, class 1 with about 8000 short text files and class 2 with about 3000 short text files. I applied LibSVM and tried a couple of parameter combinations in the cross-validation experiment.

Generally the class 1 precision falls into the range of (85%, 90%); the class 2 precision falls into the range of (70% , 75%); the recall of both class 1 and class 2 fall into the range of (80% , 85%).

For the text classification purposes, I built text feature space following the common approaches, tokening the document, filtering the stopwords and building the word vector using tf-idf or binary frequency, etc. I also tried n-gram model to build the feature space. But these approaches did not improve the performance a lot. I would like to know are there any other ways that may help tune the LibSVM to improve the performance. LibSVM provides grid search for parameter setting up, but it runs pretty slow.

Answer 1

LibSVM hasn't been getting reliable performance for me, of late. Have you tried using SVMLight ever?

You might also try looking at which features are showing the most predictive power in your model, and adding some sort of enriched-type feature. For example, if I were classifying documents on whether they contain information related to protein-protein interaction, I wouldn't really care about the specific names of the proteins as predictive features. I would pre-process my documents and normalize all protein mentions with some common term that wouldn't normally occur in my documents, like "THISWORDUSEDTOBEAPROTEIN". Previous research (sorry, I can't think of any citations off the top of my head except my own paper Ambert & Cohen, 2012) has shown that this can lead to improved classifier performance by preventing the classifier from getting distracted by common genes (e.g., ADH1A) and ignoring rare ones, instead combining the predictive power of all genes into a single feature you could think of as "GeneMentioned".

Answer 2

If you haven't done so, you can usually gain efficiency by stemming the words, as I explain on this post.

Stemming will replace words by their stem, so that for instance 'sky' and 'skies', or 'hope', 'hopes' and 'hoped', will be recognized as identical. If your texts are in English, you will most likely end up using Porter's stemming algorithm of which you can find many implementations, one of the most famous being that included in NLTK (in Python). The post mentioned above contains a code snippet that shows how to use it.

Another idea that I develop in this post is to use co-inertia analysis (CIA), a forgotten variant of canonical analysis to reduce the dimension of the data, while keeping maximal correlation with your classes. You would end up with only two or three scores instead of the few hundreds that you must have in your corpus, which might give a better bias-variance deal.

Answer 3

A simple thing to try, if you haven't done so already, is to normalize each document vector so the magnitude is 1.

SVMs tend to perform better if the magnitude of each training vector is similar.

Answer 4

You can clean up your tekst documents using Soundex, a phonetic algorithm for indexing words by sound. This Soundex methods reduces the number of spelling mistakes.

There is a good implementation of soundex in SAS, namely the function Soundex()