Return to Answer

It seems to me this question is more about "why do we keep the support vectors (because we only need know $w$ and $b$)" than "what are support vectors for".

AFAIK it's because SVMs are often used together with kernels. Without kernels, it is sufficient to store only the decision boundary $wx+b=0$, in such case the SVM will become a parametric method (instead of a nonparametric as quoted from the book).

Kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be infinite (e.g. for RBF kernels). So storing the decision boundary in such high dimensional space would be inefficient (or impossible).

If we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then the storage would be independent of the dimensionality of the implicit feature space, so that it can work with any type of kernels.

If fact if we don't use kernels, NOT to keep support vectors is more efficient in both time and space.

Say the dimension of the data is $m$ and the number of support vectors is $k$, we need $O(km)$ space to store $k$ vectors, and the time complexity for inference is also $O(km)$ because we need the inner product between the input vector and all the support vectors. Well if we only keep the parameters, the time and space complexity is both $O(m)$.

It seems to me this question is more about "why do we keep the support vectors (because after all what we want is the decision boundary)" than "what are support vectors for".

AFAIK it's because SVMs are often used together with kernels. Without kernels, it is sufficient to store only the decision boundary $wx+b=0$, and throw away the support vectors.

Kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be infinite (e.g. for RBF kernels). So using the decision boundary explicitly in such high dimensional space would be inefficient (or impossible).

If we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then the computation would depend only on the number of support vectors and the kernel function.

If fact if we don't use kernels, NOT to keep support vectors is more efficient in both time and space.

Say the dimension of the data is $m$ and the number of support vectors is $k$, we need $O(km)$ space to store $k$ vectors, and the time complexity for inference is also $O(km)$ because we need the inner product between the input vector and all the support vectors. Well if we only keep the parameters, the time and space complexity is both $O(m)$.

It seems to me that this question is more about "why do we keep the support vectors (because we only need know w and b)" than "what are the support vectors used for". AFAIK it's because SVMs are often used together with kernels.

If we don't use kernels, it is sufficient to store only the decision boundary $wx+b=0$, in such case the SVM will become a parametric method (instead of a nonparametric as quoted from the book).

Kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be infinite (e.g. for RBF kernels). So storing the decision boundary in such high dimensional space would be inefficient (or impossible).

If we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then the storage would be independent of the dimensionality of the implicit feature space, so that it can work with any type of kernels.

If fact, if we don't use kernels, NOT keeping support vectors is more efficient in both time and space.

Say the dimension of the data is $m$ and the number of support vectors is $k$, we need $O(km)$ space to store $k$ vectors, and the time complexity for inference is also $O(km)$ because we need the inner product between the input vector and all the support vectors. Well if we only keep the parameters, the time and space complexity is both $O(m)$.

It seems to me this question is more about "why do we keep the support vectors (because we only need know $w$ and $b$)" than "what are support vectors for". 

AFAIK it's because SVMs are often used together with kernels. Without kernels, it is sufficient to store only the decision boundary $wx+b=0$, in such case the SVM will become a parametric method (instead of a nonparametric as quoted from the book).

Kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be infinite (e.g. for RBF kernels). So storing the decision boundary in such high dimensional space would be inefficient (or impossible).

If we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then the storage would be independent of the dimensionality of the implicit feature space, so that it can work with any type of kernels.

If fact if we don't use kernels, NOT to keep support vectors is more efficient in both time and space.

Say the dimension of the data is $m$ and the number of support vectors is $k$, we need $O(km)$ space to store $k$ vectors, and the time complexity for inference is also $O(km)$ because we need the inner product between the input vector and all the support vectors. Well if we only keep the parameters, the time and space complexity is both $O(m)$.

AFAIK it's because SVMs are often used together with kernels.

if we don't use kernels, it is sufficient to only store the decision boundary $wx+b=0$, then SVM will become a parametric method.

kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be very high (even infinite in the case of Gaussian kernels). in this case storing the decision boundary on the high dimensional space would be inefficient (or impossible if there're infinite dimensions).

if we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then it would be independent of the dimensionality of the "implicit feature space". such that it can work with any type of kernels.

It seems to me that this question is more about "why do we keep the support vectors (because we only need know w and b)" than "what are the support vectors used for". AFAIK it's because SVMs are often used together with kernels.

If we don't use kernels, it is sufficient to store only the decision boundary $wx+b=0$, in such case the SVM will become a parametric method (instead of a nonparametric as quoted from the book).

Kernels can be thought of as mapping the input to an implicit feature space, of which the dimensionality can be infinite (e.g. for RBF kernels). So storing the decision boundary in such high dimensional space would be inefficient (or impossible).

If we decompose the decision boundary as a function of some support vectors (as shown in Daneel Olivaw's answer) then the storage would be independent of the dimensionality of the implicit feature space, so that it can work with any type of kernels.

If fact, if we don't use kernels, NOT keeping support vectors is more efficient in both time and space.

Say the dimension of the data is $m$ and the number of support vectors is $k$, we need $O(km)$ space to store $k$ vectors, and the time complexity for inference is also $O(km)$ because we need the inner product between the input vector and all the support vectors. Well if we only keep the parameters, the time and space complexity is both $O(m)$.