So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine.

However, then I tried training it on high dimensional data with few samples: Precisely speaking, I am using the complete images from the MNIST-data set, where the labels are $0$ or $1$.

I then measured performance by cross validtion (I train the SVM on 65 images, then evaluate the performance on the rest of the set) and realised the following issues:

  1. My optimal quadratic programming solution $\alpha$ is very small. That way, it is hard to decide what entry is a support vector and what is not. Every entry usually is smaller than $10^{-6}$.
  2. In the linear case detecting support vectors when $\alpha_i$ is greater than $10^{-7}$ works. I get on average 5 support vectors and an average test error of $0.00316$ (using the hard margin, no $C$ parameter). However, when introducing the soft margin by the regularization parameter $C$ and using the rbf-kernel the SVM behaves way worse. On average, the support vectors are the whole dataset and the train as well as the test error are at approximately $50\%$. I have tried changing around $C$ or $\sigma$, but the error rate does not change much.

I am sure with the rbf kernel the performance should be way better than $50\%$. I have also read that a high amount of support vectors could mean overfitting, but that does not seem to be the case here. Any help would be greatly appreciated!

Edit: I just compared with sklearn's SVM class but seem to get a similarly high error. I do not understand why rbf would perform so badly in this case compared to a linear kernel.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.