9
$\begingroup$

I'm a beginner when it comes to support vector machines. Are there some guidelines that say which kernel (e.g. linear, polynomial) is best suited for a specific problem? In my case, I have to classify webpages according to whether they contain some specific information or not, i.e. I have a binary classification problem.

Can you say in general which kernel is best suited for this task? Or do I have to try several of them on my specific dataset to find the best one? By the way, I'm using the Python library scikit-learn that makes use of the libSVM library.

$\endgroup$
  • $\begingroup$ how are you representing your webpages? bag of words? The choice of kernel depends on the similarity measure you want in your input space. $\endgroup$ – Memming Feb 3 '13 at 14:54
  • $\begingroup$ @Memming Yes, I'm using a bag-of-words representation. What do you exactly mean by similarity measure? I'm investigating whether a webpage contains some specific kind of information or not. $\endgroup$ – pemistahl Feb 3 '13 at 15:23
  • $\begingroup$ You may find this tutorial useful if you have not checked it already (csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf). Libsvm has a built-in k-fold cross validation scheme to choose between models and model parameters. $\endgroup$ – Zoran Feb 3 '13 at 20:02
  • $\begingroup$ @PeterStahl Also, it depends on what kind of class boundaries you expect. I don't have experience with bag-of-words space, so I can't help you much. $\endgroup$ – Memming Feb 3 '13 at 22:17
10
$\begingroup$

You've actually hit on something of an open question in the literature. As you say, there are a variety of kernels (e.g., linear, radial basis function, sigmoid, polynomial), and will perform your classification task in a space defined by their respective equations. To my knowledge, no one has definitively shown that one kernel always performs best on one type of text classification task versus another.

One thing to consider is that each kernel function has 1 or more parameters which will need to be optimized for your data set, which means, if you're doing it properly, you should have a second hold-out training collection on which you can investigate the best values for these parameters. (I say a second hold-out collection, because you should already have one which you are using to figure out the best input features for your classifier.) I did an experiment awhile back in which I did a large-scale optimization of each of these parameters for a simple textual classification task and found that each kernel appeared to perform reasonably well, but did so at different configurations. If I remember my results correctly, sigmoid performed the best, but did so at very specific parameter tunings--ones which took me over a month for my machine to find. My advice is that, if you have sufficient time and data to do some parameter optimization experiments, it could be interesting to compare the performance of each kernel in your particular classification task, but, if you don't, linear SVM performs reasonably well in text classification, has only the c-parameter to optimize (although many people just leaves this at default settings), and will allow you to focus on the aspects of your classification system that will have a greater contribution to final performance--the types of input features you use, and how you model them.

$\endgroup$
  • $\begingroup$ Thank you very much for your thoughts. Sorry for my late reaction. I'm writing my master thesis at the moment for which I need to evaluate results for several classifiers, not only SVM. Unfortunately, I don't have time to test many parameter combinations. That's why I will probably stick to the linear kernel. $\endgroup$ – pemistahl Feb 15 '13 at 17:05
  • $\begingroup$ Good luck! I think that's a good call. $\endgroup$ – Kyle. Feb 15 '13 at 18:54
7
$\begingroup$

Try the Gaussian kernel.

The Gaussian kernel is often tried first and turns out to be the best kernel in many applications (with your bag-of-words features, too). You should try the linear kernel, too. Don't expect it to give good results, text-classification problems tend to be non-linear. But it gives you a feeling for your data and you can be happy about how much the non-linearity improves your results.

Make sure you properly cross validate your kernel-width and think about how you want to normalize your features (tf-idf etc).

I would say you can improve your results with a better feature normalisation more than with choosing a different kernel (i.e. not the Gaussian).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.