Support vector machines and regression There's already been an excellent discussion on how support vector machines handle classification, but I'm very confused about how support vector machines generalize to regression.
Anyone care to enlighten me? 
 A: For an overview of SVM: How does a Support Vector Machine (SVM) work?
Regarding support vector regression (SVR), I find these slides from http://cs.adelaide.edu.au/~chhshen/teaching/ML_SVR.pdf  (mirror) very clear:








The Matlab documentation also has a decent  explanation and additionally goes over the optimization solving algorithm: https://www.mathworks.com/help/stats/understanding-support-vector-machine-regression.html (mirror).
So far this answer has presented the so-called epsilon-insensitive SVM (ε-SVM) regression.  There exists a more recent variant of SVM for either classification of regression: Least squares support vector machine.
Additionally, SVR may be extended for multi-output a.k.a. multi-target, e.g. see {1}.

References:


*

*{1} Borchani, Hanen, Gherardo Varando, Concha Bielza, and Pedro Larrañaga. "A survey on multi‐output regression." Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 5, no. 5 (2015): 216-233. https://scholar.google.com/scholar?cluster=10208375872303977988&hl=en&as_sdt=0,14; https://web.archive.org/web/20170628222235/http://oa.upm.es/40804/1/INVE_MEM_2015_204213.pdf
A: Basically they generalize in the same way. The kernel based approach to regression is to transform the feature, call it $\mathbf{x}$ to some vector space, then perform a linear regression in that vector space. To avoid the 'curse of dimensionality', the linear regression in the transformed space is somewhat different than ordinary least squares. The upshot is that the regression in the transformed space can be expressed as $\ell(\mathbf{x}) = \sum_i w_i \phi(\mathbf{x_i}) \cdot \phi(\mathbf{x})$, where $\mathbf{x_i}$ are observations from the training set, $\phi(\cdot)$ is the transform applied to data, and the dot is the dot product.  Thus the linear regression is 'supported' by a few (preferrably a very small number of) training vectors.  
All the mathematical details are hidden in the weird regression done in the transformed space ('epsilon-insensitive tube' or whatever) and the choice of transform, $\phi$. For a practitioner, there are also questions of a few free parameters (usually in the definition of $\phi$ and the regression), as well as featurization, which is where domain knowledge is usually helpful.
