Timeline for SVM: Number of support vectors
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 5, 2014 at 9:55 | comment | added | Marc Claesen | @DikranMarsupial Thanks for the reaction. I am aware that a large number of SVs does not guarantee an overfit, but it often is the problem when people start using the RBF kernel. I missed the fact the OP performed cross-validation, which entirely invalidated my suspicion. | |
| Dec 5, 2014 at 9:19 | answer | added | Dikran Marsupial | timeline score: 2 | |
| Dec 5, 2014 at 9:10 | comment | added | Dikran Marsupial | @MarcClaesen A large number of support vectors does not necessarily imply over-fitting. If you optimise the hyper-parameters using CV it is quite common to get a solution with a very bland kernel and a small value of C, in which case you end up with a lot of the data being support vectors, but a smooth model. I wouldn't worry about it, the sparsity of SVMs is a nice by-product, but little more than that - I use LS-SVMs for this kind of thing, which are fully dense, but just as good in terms of generalisation. | |
| Dec 5, 2014 at 6:32 | comment | added | NOP | And actually, no the kernel does not look like a unit matrix. | |
| Dec 5, 2014 at 6:13 | comment | added | NOP | Probably also worth adding it is a reasonably small dataset (only ~400 examples in total, classes are also fairly uneven). I'm using leave-one-subject-out crossvaliation and getting around 93-97% on average. | |
| Dec 5, 2014 at 6:03 | comment | added | Marc Claesen | Usually a very high number of support vectors indicates you are overfitting. I am going to guess your kernel matrix looks like the unit matrix. This is caused by using a too low bandwidth for the RBF kernel (e.g. low $\sigma$ or high $\gamma$, depending on your parameterization). What is your cross-validation performance? | |
| Dec 5, 2014 at 5:52 | review | First posts | |||
| Dec 5, 2014 at 6:24 | |||||
| Dec 5, 2014 at 5:47 | history | asked | NOP | CC BY-SA 3.0 |