# Support vector machines and regression

There's already been an excellent discussion on how support vector machines handle classification, but I'm very confused about how support vector machines generalize to regression.

Anyone care to enlighten me?

Basically they generalize in the same way. The kernel based approach to regression is to transform the feature, call it $\mathbf{x}$ to some vector space, then perform a linear regression in that vector space. To avoid the 'curse of dimensionality', the linear regression in the transformed space is somewhat different than ordinary least squares. The upshot is that the regression in the transformed space can be expressed as $\ell(\mathbf{x}) = \sum_i w_i \phi(\mathbf{x_i}) \cdot \phi(\mathbf{x})$, where $\mathbf{x_i}$ are observations from the training set, $\phi(\cdot)$ is the transform applied to data, and the dot is the dot product. Thus the linear regression is 'supported' by a few (preferrably a very small number of) training vectors.

All the mathematical details are hidden in the weird regression done in the transformed space ('epsilon-insensitive tube' or whatever) and the choice of transform, $\phi$. For a practitioner, there are also questions of a few free parameters (usually in the definition of $\phi$ and the regression), as well as featurization, which is where domain knowledge is usually helpful.

• From an intuition viewpoint, is this almost like a single-class classification, where the class "boundary" line ends up running through the points instead of between the points of two classes? – Wayne Aug 17 '11 at 18:01
• @Wayne, that's my understanding, yes. I'm not 100%, however. – Zach Jan 27 '12 at 22:14

For an overview of SVM: How does a Support Vector Machine (SVM) work?

Regarding support vector regression (SVR), I find these slides from http://cs.adelaide.edu.au/~chhshen/teaching/ML_SVR.pdf (mirror) very clear:

The Matlab documentation also has a decent explanation and additionally goes over the optimization solving algorithm: https://www.mathworks.com/help/stats/understanding-support-vector-machine-regression.html (mirror).

So far this answer has presented the so-called epsilon-insensitive SVM (ε-SVM) regression. There exists a more recent variant of SVM for either classification of regression: Least squares support vector machine.

Additionally, SVR may be extended for multi-output a.k.a. multi-target, e.g. see {1}.

References: