28
$\begingroup$

There's already been an excellent discussion on how support vector machines handle classification, but I'm very confused about how support vector machines generalize to regression.

Anyone care to enlighten me?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
19
$\begingroup$

Basically they generalize in the same way. The kernel based approach to regression is to transform the feature, call it $\mathbf{x}$ to some vector space, then perform a linear regression in that vector space. To avoid the 'curse of dimensionality', the linear regression in the transformed space is somewhat different than ordinary least squares. The upshot is that the regression in the transformed space can be expressed as $\ell(\mathbf{x}) = \sum_i w_i \phi(\mathbf{x_i}) \cdot \phi(\mathbf{x})$, where $\mathbf{x_i}$ are observations from the training set, $\phi(\cdot)$ is the transform applied to data, and the dot is the dot product. Thus the linear regression is 'supported' by a few (preferrably a very small number of) training vectors.

All the mathematical details are hidden in the weird regression done in the transformed space ('epsilon-insensitive tube' or whatever) and the choice of transform, $\phi$. For a practitioner, there are also questions of a few free parameters (usually in the definition of $\phi$ and the regression), as well as featurization, which is where domain knowledge is usually helpful.

Improve this answer
$\endgroup$
3
  • $\begingroup$ From an intuition viewpoint, is this almost like a single-class classification, where the class "boundary" line ends up running through the points instead of between the points of two classes? $\endgroup$
    – Wayne
    Aug 17, 2011 at 18:01
  • $\begingroup$ @Wayne, that's my understanding, yes. I'm not 100%, however. $\endgroup$
    – Zach
    Jan 27, 2012 at 22:14
  • $\begingroup$ I am confused by this answer. To explain how SVM or SVR works, for the linear case, no kernel method is involved. Shouldn't we be able to explain the relationship between SVM and SVR without talking about the kernel method ? $\endgroup$
    – John Smith
    Aug 23, 2022 at 20:07
9
$\begingroup$

For an overview of SVM: How does a Support Vector Machine (SVM) work?

Regarding support vector regression (SVR), I find these slides from http://cs.adelaide.edu.au/~chhshen/teaching/ML_SVR.pdf (mirror) very clear:

enter image description here

enter image description hereenter image description here

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

The Matlab documentation also has a decent explanation and additionally goes over the optimization solving algorithm: https://www.mathworks.com/help/stats/understanding-support-vector-machine-regression.html (mirror).

So far this answer has presented the so-called epsilon-insensitive SVM (ε-SVM) regression. There exists a more recent variant of SVM for either classification of regression: Least squares support vector machine.

Additionally, SVR may be extended for multi-output a.k.a. multi-target, e.g. see {1}.

References:

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.