# Support vector machines and regression

There's already been an excellent discussion on how support vector machines handle classification, but I'm very confused about how support vector machines generalize to regression.

Anyone care to enlighten me?

## 2 Answers

Basically they generalize in the same way. The kernel based approach to regression is to transform the feature, call it $\mathbf{x}$ to some vector space, then perform a linear regression in that vector space. To avoid the 'curse of dimensionality', the linear regression in the transformed space is somewhat different than ordinary least squares. The upshot is that the regression in the transformed space can be expressed as $\ell(\mathbf{x}) = \sum_i w_i \phi(\mathbf{x_i}) \cdot \phi(\mathbf{x})$, where $\mathbf{x_i}$ are observations from the training set, $\phi(\cdot)$ is the transform applied to data, and the dot is the dot product. Thus the linear regression is 'supported' by a few (preferrably a very small number of) training vectors.

All the mathematical details are hidden in the weird regression done in the transformed space ('epsilon-insensitive tube' or whatever) and the choice of transform, $\phi$. For a practitioner, there are also questions of a few free parameters (usually in the definition of $\phi$ and the regression), as well as featurization, which is where domain knowledge is usually helpful.

• From an intuition viewpoint, is this almost like a single-class classification, where the class "boundary" line ends up running through the points instead of between the points of two classes? Commented Aug 17, 2011 at 18:01
• @Wayne, that's my understanding, yes. I'm not 100%, however.
– Zach
Commented Jan 27, 2012 at 22:14
• I am confused by this answer. To explain how SVM or SVR works, for the linear case, no kernel method is involved. Shouldn't we be able to explain the relationship between SVM and SVR without talking about the kernel method ? Commented Aug 23, 2022 at 20:07

For an overview of SVM: How does a Support Vector Machine (SVM) work?

Regarding support vector regression (SVR), I find these slides from http://cs.adelaide.edu.au/~chhshen/teaching/ML_SVR.pdf (mirror) very clear:

The Matlab documentation also has a decent explanation and additionally goes over the optimization solving algorithm: https://www.mathworks.com/help/stats/understanding-support-vector-machine-regression.html (mirror).

So far this answer has presented the so-called epsilon-insensitive SVM (ε-SVM) regression. There exists a more recent variant of SVM for either classification of regression: Least squares support vector machine.

Additionally, SVR may be extended for multi-output a.k.a. multi-target, e.g. see {1}.

References: