I understand when the data is not linearly separable, it has to transformed into higher dimensional space, to make it linearly separable. Applying kernel trick can perform it without even computing the co-ordinates of the points in that higher dimension.

But I failed to figure out, what is the end result we get after applying kernel trick.

I saw a scary looking vector, results from a dot product ->

which looks like, (a^T . b)^2, where a and b are two vectors.

What do we get after applying the kernel trick?

Is it the equation of the hyperplane or coordiantes of the points ?


1 Answer 1


Kernel functions fundamentally make two pictures equivalent:

  1. The feature map picture: A high dimensional dot product (ie a linear operation in very high dim)
  2. The kernel function picture: A low-dimensional nonlinear mathematical operation

So it's hard to make a rule to visualize these operations by rule: intuition is maybe best gained on a case-by-case basis.

In your example for vectors $a \in \mathbb{R}^n$ and $b \in \mathbb{R}^n$, the kernel is the square of the dot product, and we can represent it as a dot product of a higher dimension:

($a^\top b)^2 = (\sum_{i=1}^{n}a_ib_i)^2 = (\sum_{i=1}^{n}a_ib_i)(\sum_{j=1}^{n}a_jb_j)$

Where we make sure to use different indices for the sums so as to not confuse ourselves with the summation behavior. First distribute the left sum into the right:

($a^\top b)^2 = \sum_{i=1}^{n}(a_ib_i\sum_{j=1}^{n}a_jb_j)$

Now distribute each $a_ib_i$ into the right sum and group similar terms:

($a^\top b)^2 = \sum_{i=1}^{n}\sum_{j=1}^{n}(a_ib_ia_jb_j) = \sum_{i=1}^{n}\sum_{j=1}^{n}(a_ia_j)(b_ib_j)$

Now, the tricky part: think about what this double sum is doing. It's taking a sum (over two sets of indices) of a bunch of simple multiplications. If we defocus from the two sets of indices, that's exactly what a dot product is: a sum of simple multiplications. It's just that we need to do the dot product over $n\times n = n^2$ dimension, and we're just multiplying every component $a_i$ with every component $a_j$ and doing the same for the other components of $b_i$ and $b_j$. Finally:

$ (a^\top b)^2 = \phi(a)^\top \phi(b) $

Where the feature map for a is:

$\phi(a) = \begin{bmatrix} a_1a_1 & a_1a_2 & ... & a_1a_n & a_2a_1 & ... & a_2a_n & ... & a_n a_n \end{bmatrix}^\top $

Kind of challenging to write, but it's a ton of multiplied components. Maybe not the easiest to visualize. Better to think of the original dot product!

$a^\top b$ does two two things: it gets small when the vectors are orthogonal or if their magnitudes are small. The dot product is negative if the vectors are opposite in direction. So taking the square vaguely measures how far are the vectors from being orthogonal: if they point towards each other or away from each other. Also, it scales magnitudes as the square, so squaring that dot product gives heavy weight to high magnitude vectors.

  • $\begingroup$ What the final result tells us ?...Does it gives the equation or position of the Hyperplane ? $\endgroup$ Commented Jul 29, 2023 at 7:47
  • 1
    $\begingroup$ @mainakmukherjee The final feature map dot product is just a mathematical equivalency in equations-- there's no separating hyperplane in a kernel mapping itself. The separating hyperplane comes from a ML problem you apply. eg you may take all of your datapoints in $a$ and $b$ and find their kernelized vector form (ie feature maps $\phi(a)$ and $\phi(b)$. So now it's as if you had each component of the feature vector on its own axis, and you can apply your ML model to plot a separating hyperplane in that space. That's equivalent to a nonlinear hyperplane in the normal $a$ and $b$ vector spaces. $\endgroup$ Commented Jul 29, 2023 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.