# Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear

I am interested in learning the math behind support vector machines. So far, I understand that SVMs attempt to find hyperplanes that maximize the margin distance between support vectors associated with the classes that you are trying to classify. I also understand that using a kernel allows one to project the data points onto higher dimensions to help classify data that is not linearly separable. I am playing around with an SVM using the Scikitlearn Python library and I found that the .coef_ attribute returns the normal vectors associated with the hyperplanes that are calculated in the fitting procedure. I am interested in looking at these normal vectors to get a sense of how much each feature contributes to calculating the hyperplanes. However, it seems that the normal vectors are only available if you use a linear SVM. I don't see why the normal vectors wouldn't be available if a kernel is used. From what I understand, you are just finding hyperplanes in a space with more dimensions than the number of features in your original dataset. Shouldn't you be able to get the normal vectors to those hyperplanes as well? What am I missing here?

I fully appreciate any insight into this question as I am still a novice when it comes to the math behind a lot of machine learning techniques.

Thanks!

Your intuition is on the right track. The usage of kernels is what makes SVMs so nice/convenient, since (very informally) it gives us the ability to build a classifier in some higher-dimensional space implicitly without having to operate in that higher-dimensional space, so I can understand why you'd have this question.

The short answer is, theoretically, sure, you could recover the coefficients defining your separating hyperplane if, given your feature map $$\phi: \mathbb{R}^n \to \mathbb{R}^p$$, you know the explicit form of $$\phi$$. As an illustrative example, consider the polynomial kernel of degree 2, i.e., $$K(x^{(i)}, x^{(j)}) = (x^{(i)\top} x^{(j)})^2$$ for some $$x^{(i)}, x^{(j)} \in \mathbb{R}^n$$. Some algebra shows that

$$(x^{(i)\top} x^{(j)})^2 = \sum_{k=1}^n \sum_{\ell=1}^n (x^{(i)}_kx^{(i)}_\ell)(x^{(j)}_k x^{(j)}_\ell),$$

which is an inner product between two feature maps of $$x^{(i)}, x^{(j)}$$. That is, $$K(x^{(i)}, x^{(j)})$$ can be written in the form $$\phi(x^{(i)})^\top \phi(x^{(j)})$$, where $$d = n^2$$ for this case. Further examples of such kernels can be found here, as well as criterion for what constitutes a "valid" kernel.

In practice, I'm not completely sure what the use of seeing the hyperplane coefficients in that higher-dimensional space is. But if you really, really want the coefficients for your hyperplane in that higher dimensional space, then they can be computed in terms of the coefficients of the dual optimization problem. That is; for coefficients $$w, x^{(i)} \in \mathbb{R}^n$$, class labels $$y^{(i)} \in \{-1, +1\}$$, and dual variables $$\alpha^{(i)} \in \mathbb{R}^+$$, and $$i \in \{1, \dots, M\}$$:

$$w = \sum_{i=1}^M \alpha^{(i)} y^{(i)} \phi(x^{(i)})$$

The derivation follows from taking the dual of the primal SVM objective, constructing the Lagrangian, and minimizing in terms of the primal variables (i.e., the hyperplane coefficients) for some fixed $$\alpha^{(i)}$$.

However, some feature maps are such that $$p = \infty$$ (i.e., your kernel projects features into an "infinite dimensional" space, which is called a Hilbert space), such as the RBF/Gaussian kernel, which makes it a bit hard to compute the above. To see how this is possible, consider the Gaussian kernel $$K(x^{(i)}, x^{(j)}) \triangleq \exp(-\frac{1}{2}\lVert x^{(i)} - x^{(j)}\rVert_2^2).$$ We can expand the squared norm term and, recalling that $$\exp(z) = \sum_{k=0}^\infty z^k/k!$$, we end up with something like $$\sum_{k=0}^\infty \frac{(x^{(i)\top} x^{(j)})^k}{k!} \exp(-\lVert x^{(i)} \rVert_2^2 / 2) \exp(-\lVert x^{(j)} \rVert_2^2 / 2).$$

Using the multinomial theorem to expand the $$(x^{(i)\top} x^{(j)})^k$$ term, you can verify that we end up with a valid inner product, giving us a kernel. Alternately, you can analyze each $$k$$ separately, and, noticing that each term of the infinite summation for fixed $$k$$ is a rescaled polynomial kernel of degree $$k$$, interpret the RBF kernel as an infinite sum of polynomial kernels. Thus, we can't really recover the true hyperplane parameters. If you haven't seen this before, it might be surprising that you can (informally) implicitly work in an infinite-dimensional feature space.

• Thank you for your explaination! I need to read more about the dual optimization problem and Hilbert spaces to figure out if recovering the hyperplane weights is feasible/possible for the problem I'm working on. Commented Aug 14, 2023 at 2:05
• I don't think you need to know the details of the dual problem for your question, though it would certainly supplement your understanding of SVMs. The feasibility of recovering the weights should solely depend on the kernel used, since it specifies your feature map, right? I'm still unsure why you need the hyperplane parameters since the image o the feature fmap isn't inherently "meaningful" per se, but feel free to pose a new question if I'm missing something. Commented Aug 14, 2023 at 2:58

What you're missing here is that the representation of kernels as the sums of nonlinear features is not known. What's known is that these kernel are likely to be representable as those new features, so we use them.

Details. The dual problem involves minimizing a quantity $$\min_\alpha \left[\sum_{ij}y_i\alpha_i\left(\sum_k\phi_k(x_i)\phi_k(x_j)\right)\alpha_jy_j+\dots\right]$$ Where $$\phi_k(x)$$ is a set of nonlinear features created on the data $$X$$. So, for some sets of these features it is possible to capture these sums with kernels $$K(x_i,x_j)=\sum_k\phi_k(x_i)\phi_k(x_j)$$. Then the dual problem becomes $$\min_\alpha \left[\sum_{ij}y_i\alpha_iK(x_i,x_j)\alpha_jy_j+\dots\right]$$

Once the kernel is found we don't need the new features $$\phi(x)$$ themselves and can proceed working with just a kernel. Moreover, we don't even need to know these new features, and as a matter of fact we usually don't know them. We know the conditions on the functions K(.,.) that would make them the valid kernels, so we can almost arbitrarily try functions as kernels so long they satisfy the conditions, then we don't bother looking for the features.

The reason for using kernels is that they represent rich set of new nonlinear features but can be relatively fast to calculate. As you saw the kernel represents a dot product $$\sum_k\phi_k(x_i)\phi_k(x_j)$$ where $$\phi_k(.)$$ is often an infinite set. Instead of messing with dot products of infinite sets of function that potentially can be expensive to compute we get the result much faster with a kernel.