1
$\begingroup$

Consider the case where more than the minimal number of vectors lie on the lines (or hyperplanes in higher dimensions) defined by the margin found by the SVM algorithm. For example, see the image below where a third redundant vector lies on the margin. Do we have any guarantees on the selection of support vectors in this case? For example, will all three points be stored as support vectors or just the minimal number necessary? If it's the former, is there any structure in their associated dual coefficients?

One of the three blue "support vectors" here is redundant. How does the SVM deal with this?

$\endgroup$

1 Answer 1

1
$\begingroup$

No, quite often you end up with redundant support vectors that lie on the margin, but the decision surface can be constructed with only a subset of the identified support vectors. The example you give with three co-linear support vectors is a good example of that. Vapnik gave a name to the support vectors without which you can't construct the decision surface, but I can't remember what it is off hand.

Whether the training algorithm will identify all of the support vectors or just those necessary to represent the decision surface is likely to vary from one algorithm to another. In this example, an algorithm that is based on identifying the closest points between convex hulls of the examples belonging to each class may well only identify two of the three support vectors. One based on Sequential Minimal Optimisation would probably depend on the way it identifies the next dual parameter to optimise, and something like interior point may identify all three?

$\endgroup$
1
  • $\begingroup$ Please do add to the answer if you remember the name for such support vectors, it would be interesting to know their name. So from my reading of your answer there does exist a method by which you could keep only the minimal number of support vectors without much additional overhead? $\endgroup$
    – Seraf Fej
    Jan 16, 2022 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.