How kernel functions are related to the similarity between two vectors?

Question

I was reading about regression with Gaussian processes and I bumped into kernel functions. These functions can be expressed as inner products in a new feature space $\mathcal{M}$, that is:

$$k(\mathbf{x}, \mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle_{\mathcal{M}}$$

with $\phi$ a feature map from the original $\mathcal{X}$ to the new feature space:

$$\phi: \mathcal{X} \times \mathcal{X} \rightarrow \mathcal{V}$$

Lets take an example of a kernel function, the RBF kernel, which is defined as:

$$\exp\left(-\gamma\lvert \mathbf{x} - \mathbf{y}\rvert^2\right)$$

Is it correct to say that this function measures the similarity of the two vectors $\mathbf{x}$ and $\mathbf{y}$ in the original space and not in the new space, even though the inner product is performed in the new space? I am asking this question because sometimes the cosine similarity, which is related with the inner product in the original space, is used as a similarity measure (where there is no need to transform in a new space).

Finally, are statements such as:

Inner product measures the similarity between two vectors.

misnomers?

I checked Scikit-learn where it is stated that:

1. Distance metrics are functions d(a, b) such that d(a, b) < d(a, c) if objects a and b are considered “more similar” than objects a and c.
2. Kernels are measures of similarity, i.e. s(a, b) > s(a, c) if objects a and b are considered “more similar” than objects a and c.

Is it correct to say that kernels are the inverses of distance metrics? That is lower distance means greater similarity? For Gaussian kernel it is easy to see it (similarity increases when the Euclidean distance decreases). What about other kernels?

Take the linear kernel as an example, which is nothing else than the inner product in the original space. What distance is reduced when the similarity increases? Clearly, the distance can't be Euclidean distance. This can be justified by considering three vectors $\mathbf{a}, \mathbf{b}, \mathbf{c}$ where $\mathbf{b} = 10000\mathbf{c}$.

Answer 1

The kernel $k(x,y), x, y\in \mathbb R^n$, if it satisfies certain conditions, is the scalar product of $\Phi(x)$ and $\Phi(y)$ in some vector space $S$, $\Phi: \mathbb{R}^n\to S$. And the scalar product is often interpreted as a measure of similarity. Then, $k(x,y)$ would be a measure for how similar $\Phi(x)$ and $\Phi(y)$ are. Furthermore, this similarity between $\Phi(x)$ and $\Phi(y)$ is then interpreted as a measure of similarity between the original $x$ and $y$.

And you are right: this measure of similarity via scalar products and the measure of similarity via the inverse of a distance (e.g. Euclidean), are not very compatible. You can have an arbitrarily large scalar product between two arbitrarily distant points $x$ and $y$, and you can have an arbitrarily small scalar product between two identical points.

Those two notions would be compatible if we restricted the points to a sphere (all vectors have the same length) or consider them in projective space. In the case of Gaussian kernels: $$ k_G(x,y) := \exp(\gamma \|x-y\|), $$ all $\Phi(x), x\in\mathbb R^n$ have norm one: $$ \|\Phi(x)\|^2 = k(x, x) = \exp(\gamma \|x - x\|) = \exp(0) = 1. $$ Thus, in this common case, fortunately, inverse distance and the scalar product are quite compatible as similarity notions. On the other hand, your example with the linear kernel is clearly an example of when they are not.

Answer 2

Yes, distance is the opposite of similarity. Not every distance or similarity metric has a named counterpart. To convince yourself, use the definitions you referred to, they differ only by swapping $<$ and $>$.