Kernel functions fundamentally make two pictures equivalent:
- The feature map picture: A high dimensional dot product (ie a linear operation in very high dim)
- The kernel function picture: A low-dimensional nonlinear mathematical operation
So it's hard to make a rule to visualize these operations by rule: intuition is maybe best gained on a case-by-case basis.
In your example for vectors $a \in \mathbb{R}^n$ and $b \in \mathbb{R}^n$, the kernel is the square of the dot product, and we can represent it as a dot product of a higher dimension:
($a^\top b)^2 = (\sum_{i=1}^{n}a_ib_i)^2 = (\sum_{i=1}^{n}a_ib_i)(\sum_{j=1}^{n}a_jb_j)$
Where we make sure to use different indices for the sums so as to not confuse ourselves with the summation behavior. First distribute the left sum into the right:
($a^\top b)^2 = \sum_{i=1}^{n}(a_ib_i\sum_{j=1}^{n}a_jb_j)$
Now distribute each $a_ib_i$ into the right sum and group similar terms:
($a^\top b)^2 = \sum_{i=1}^{n}\sum_{j=1}^{n}(a_ib_ia_jb_j)
= \sum_{i=1}^{n}\sum_{j=1}^{n}(a_ia_j)(b_ib_j)$
Now, the tricky part: think about what this double sum is doing. It's taking a sum (over two sets of indices) of a bunch of simple multiplications. If we defocus from the two sets of indices, that's exactly what a dot product is: a sum of simple multiplications. It's just that we need to do the dot product over $n\times n = n^2$ dimension, and we're just multiplying every component $a_i$ with every component $a_j$ and doing the same for the other components of $b_i$ and $b_j$. Finally:
$ (a^\top b)^2 = \phi(a)^\top \phi(b) $
Where the feature map for a is:
$\phi(a) = \begin{bmatrix}
a_1a_1 & a_1a_2 & ... & a_1a_n & a_2a_1 & ... & a_2a_n & ... & a_n a_n
\end{bmatrix}^\top
$
Kind of challenging to write, but it's a ton of multiplied components. Maybe not the easiest to visualize. Better to think of the original dot product!
$a^\top b$ does two two things: it gets small when the vectors are orthogonal or if their magnitudes are small. The dot product is negative if the vectors are opposite in direction. So taking the square vaguely measures how far are the vectors from being orthogonal: if they point towards each other or away from each other. Also, it scales magnitudes as the square, so squaring that dot product gives heavy weight to high magnitude vectors.
