I have about 30000 book names assigned to 6 categories, and I want to build scalable and accurate classifiers. So far I have only been able to use Naive Baye's and LibLINEAR classifiers and they both give me an (almost) identical precision and recall values of 0.8 and 0.7 after 10 fold CV.

I am wondering if I would be able to do better if I were to use more complex models . The problem is that the time complexity of the sophisticated models seems to increase super-linearly with the number of training instances. SVM (SMO implementation from WEKA), for example, has been running for the past 3hrs already on this data, whereas the Naive Baye's and LibLINEAR finished in about 15mins and 40mins respectively.

I am trying to build a general framework for short texts classification (twitter, text messages etc.), and so will be running many experiments over varied data sets. I require techniques that scale and work well (don't we all :-)). Any suggestions?

Another question is with regard to dimension reduction. When I pre-process my text, I apply stemming, stopword removal and convert the text to tf-idf vector representation. Dimension reduction techniques (Info gain, in particular) again seems to be taking an inordinately long time. Any scalable way to do feature selection? Would pruning by tf-idf scores an acceptable approach?

Edit 1: By "Info Gain", I meant Information Gain . And currently I am not doing any feature selection.

  • 2
    $\begingroup$ I'm not sure that topic modeling is quite what you're looking for, but it seems like it ought to at least be on your radar if you're working on text classification problems. David Blei has some good resources. $\endgroup$ – Matt Parker Dec 1 '11 at 19:58
  • $\begingroup$ LibLinear just released a LibShortText package csie.ntu.edu.tw/~cjlin/libshorttext $\endgroup$ – Erel Segal-Halevi Jul 3 '13 at 13:18
  • $\begingroup$ tf-idf might not be very useful transformation for short texts. Stop word removal might remove too much information. $\endgroup$ – Vladislavs Dovgalecs Feb 24 '16 at 20:10

Is your SVM implementation parallel? One simple idea would be to split your 10-fold CV across 10 machines (or cores). This should reduce the algorithm's runtime to almost 1/10 of its current running time.

What do you mean by "info gain?" Have you tried applying LibLINEAR on a dataset with no features removed?

  • $\begingroup$ +1 That's a good suggestion. My current SVM implementation is not parallel. $\endgroup$ – user721975 Dec 1 '11 at 20:14
  • $\begingroup$ Edited my OP to answer your other questions. $\endgroup$ – user721975 Dec 1 '11 at 20:15

The first thing you need to do is to figure out is the source of your generalization error. Is it "bias" or is it "variance"? (or perhaps something else?) If it is variance, your training set might be small for what you are trying to accomplish, and you might need more training data. If it is bias, then changing to a different model, or changing the parameters of your SVM might help you get a better result.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.