# SVM: Number of support vectors

Imagine I am using an SVM to train a classifier for a given dataset, in one-vs-all configuration. For each class, I am performing cross validation for parameter selection (grid search to choose parameters for both SVM and kernel). I am also using the RBF kernel.

In many cases, what I am observing is that the number of support vectors for the positive class exactly matches (or is very close to) the total number of positive examples. The support vectors for the negative class vary, but is still quite high.

Thus, my questions are:

1) For a linear case, is it reasonable to expect that the number of support vectors from the positive set would be significantly less (given that the data is apparently not linearly seperable).

2) Does the number of support vectors selected reflect classification "capacity" in the sense that a high capacity classifier can separate highly overlapping classes with a complex decision boundary? Is it reasonable to assume that a higher number of support vectors is indicative of high capacity?

• Usually a very high number of support vectors indicates you are overfitting. I am going to guess your kernel matrix looks like the unit matrix. This is caused by using a too low bandwidth for the RBF kernel (e.g. low $\sigma$ or high $\gamma$, depending on your parameterization). What is your cross-validation performance? Dec 5, 2014 at 6:03
• Probably also worth adding it is a reasonably small dataset (only ~400 examples in total, classes are also fairly uneven). I'm using leave-one-subject-out crossvaliation and getting around 93-97% on average.
– NOP
Dec 5, 2014 at 6:13
• And actually, no the kernel does not look like a unit matrix.
– NOP
Dec 5, 2014 at 6:32
• @MarcClaesen A large number of support vectors does not necessarily imply over-fitting. If you optimise the hyper-parameters using CV it is quite common to get a solution with a very bland kernel and a small value of C, in which case you end up with a lot of the data being support vectors, but a smooth model. I wouldn't worry about it, the sparsity of SVMs is a nice by-product, but little more than that - I use LS-SVMs for this kind of thing, which are fully dense, but just as good in terms of generalisation. Dec 5, 2014 at 9:10
• @DikranMarsupial Thanks for the reaction. I am aware that a large number of SVs does not guarantee an overfit, but it often is the problem when people start using the RBF kernel. I missed the fact the OP performed cross-validation, which entirely invalidated my suspicion. Dec 5, 2014 at 9:55