What is the Reproducing Kernel Hilbert Space of $k(x,y)=\langle x,y\rangle^2$?

Question

Digging in the definition of Reproducing Kernel Hilbert Spaces (RKHS) I came across the following example taken from pages 49-51 of [1]:

Given the kernel $k(x,y) = \langle x,y\rangle^2$, with $x,y\in \mathbb R^2$, one can show that both

$\phi(x) = [x_1^2,~ x_2^2,~ x_1x_2, ~x_1x_2]$, with feature space $\mathcal H = \mathbb R^4$

$\phi^{\prime}(x) = [x_1^2,~ x_2^2,~ \sqrt 2 x_1x_2]$, with feature space $\mathcal H^\prime = \mathbb R^3$

are valid feature maps for the kernel. However, the slides point out that neither $\mathcal H$ nor $\mathcal H^\prime$ are RKHS because the RKHS must by definition be unique for a given kernel. Can you explain what the RKHS would be in this case?

Other references on the topic state that the RKHS $\mathcal H_k$ is unique up to isomorphism [2]. Under this definition I guess both $\mathcal H$ and $\mathcal H^\prime$ can be considered RKHS by being isomorphic to $\mathcal H_k$.

[1] http://www.stats.ox.ac.uk/~sejdinov/teaching/atml14/Theory_slides1_2014.pdf

[2] http://users.umiacs.umd.edu/~hal/docs/daume04rkhs.pdf

Answer 1

Prof. Dino Sejdinovic, author of the slides my question referred to, was kind enough to send me the following answer:

Feature spaces are not unique and indeed they are all isomorphic. RKHS of a given kernel is unique as a space of functions, i.e. the one that contains functions of the form $k(\cdot , x)$. If these functions form a finite-dimensional RKHS, that RKHS is isomorphic to a standard Euclidean space of that dimension. For intuition about uniqueness for infinite-dimensional case, you can consider an orthonormal basis of $\mathcal H_k$, e.g. given by Mercer's theorem -- then the $\ell^2$ space of coefficients corresponding to this basis is a valid feature space and isomorphic to RKHS. You can choose many different bases so there are many versions of feature spaces. But they all correspond to the same space of functions which is the RKHS.

In addition, the first twenty minutes of this class by Prof. Lorenzo Rosasco delve a bit deeper into the relation between feature spaces, feature maps and RKHS.