Which SVM kernel to use for a binary classification problem?

Question

I'm a beginner when it comes to support vector machines. Are there some guidelines that say which kernel (e.g. linear, polynomial) is best suited for a specific problem? In my case, I have to classify webpages according to whether they contain some specific information or not, i.e. I have a binary classification problem.

Can you say in general which kernel is best suited for this task? Or do I have to try several of them on my specific dataset to find the best one? By the way, I'm using the Python library scikit-learn that makes use of the libSVM library.

Answer 1

You've actually hit on something of an open question in the literature. As you say, there are a variety of kernels (e.g., linear, radial basis function, sigmoid, polynomial), and will perform your classification task in a space defined by their respective equations. To my knowledge, no one has definitively shown that one kernel always performs best on one type of text classification task versus another.

One thing to consider is that each kernel function has 1 or more parameters which will need to be optimized for your data set, which means, if you're doing it properly, you should have a second hold-out training collection on which you can investigate the best values for these parameters. (I say a second hold-out collection, because you should already have one which you are using to figure out the best input features for your classifier.) I did an experiment awhile back in which I did a large-scale optimization of each of these parameters for a simple textual classification task and found that each kernel appeared to perform reasonably well, but did so at different configurations. If I remember my results correctly, sigmoid performed the best, but did so at very specific parameter tunings--ones which took me over a month for my machine to find. My advice is that, if you have sufficient time and data to do some parameter optimization experiments, it could be interesting to compare the performance of each kernel in your particular classification task, but, if you don't, linear SVM performs reasonably well in text classification, has only the c-parameter to optimize (although many people just leaves this at default settings), and will allow you to focus on the aspects of your classification system that will have a greater contribution to final performance--the types of input features you use, and how you model them.

Answer 2

Try the Gaussian kernel.

The Gaussian kernel is often tried first and turns out to be the best kernel in many applications (with your bag-of-words features, too). You should try the linear kernel, too. Don't expect it to give good results, text-classification problems tend to be non-linear. But it gives you a feeling for your data and you can be happy about how much the non-linearity improves your results.

Make sure you properly cross validate your kernel-width and think about how you want to normalize your features (tf-idf etc).

I would say you can improve your results with a better feature normalisation more than with choosing a different kernel (i.e. not the Gaussian).