Regression with a kernel

Question

I have a fixed kernel and a set of points. I do SVC with the flavor of SVM classification i'm working on (assume it's just a regular SVM) and i obtain a classifier represented by an explicit vector of coefficients and a threshold. This could be with any kernel.

Then i need to generate a regressor for those points using the same kernel i've used for the SVM. This is easy: i use a SVR and it's all good.

What i need though is the angle between my classifier and the regressor. Why i need that would take too much time to explain. To calculate the angle i need the explicit vector of coefficients of the SVR but this can be calculated only for linear kernels because it would require the explicit mapping otherwise.

What i want to ask is: can i calculate the angle between my regressor and my classifier without obtaining the explicit vector of coefficients? Otherwise, is there a way to approximate the mapping for any given kernel so that i can calculate the explicit vector of coefficients using the dual formulation of the SVR problem?

Answer 1

I am not sure I understand your terminology. I assume that by "vector of coefficients", you mean $w$ as used here. I guess you want an angle between the two separating hyperplanes (or normal vectors, to be precise), for some mysterious reason.

To calculate the angle i need the explicit vector of coefficients of the SVR but this can be calculated only for linear kernels because it would require the explicit mapping otherwise.

You can still compute an angle without an explicit mapping. Assume that you have $n$ training points. Then your solution from SVC is a sparse vector $\alpha = (\alpha_1, \ldots, \alpha_n)^\top$ and your solution from SVR is $\beta = (\beta_1, \ldots, \beta_n)^\top$. The normal vector obtained from SVC, denoted by $f$, is an element in RKHS $\mathcal{H}$ associated with your kernel $k$, having the form

$f = \sum_{i=1}^n \alpha_i y_i k(\cdot, x_i)$

where $y_i \in \{-1, 1\}$.

The normal vector from SVR denoted by $g \in \mathcal{H}$ is

$g = \sum_{i=1}^n \beta_i k(\cdot, x_i)$.

An angle $\theta$ between two vectors $f$ and $g$ (see this page) is given by

$\theta = \arccos \frac{\langle f, g \rangle_\mathcal{H}}{\|f\|_\mathcal{H} \|g\|_\mathcal{H}}.$

The inner product part is

$ \begin{align*} \langle f, g \rangle_\mathcal{H} &= \langle \sum_{i=1}^n \alpha_i y_i k(\cdot, x_i), \sum_{j=1}^n \beta_j k(\cdot, x_j) \rangle_\mathcal{H} \\ &= \sum_{i=1}^n \sum_{j=1}^n \alpha_i y_i \beta_j k(x_i, x_j) \end{align*} $

The norm is given by

$ \begin{align*} \|f\| &= \sqrt{\langle f, f \rangle} = \sqrt{\sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j k(x_i, x_j)} \end{align*}. $

The norm for $g$ which is $\|g\|$ is

$ \begin{align*} \|g\| &= \sqrt{\langle g, g \rangle} = \sqrt{\sum_{i=1}^n \sum_{j=1}^n \beta_i \beta_j k(x_i, x_j)} \end{align*}. $

You have all you need to compute $\theta$. The number of training samples in for the two predictors can be different. But the kernel $k$ must be the same because it makes no sense to calculate an angle of vectors from two different spaces.