About Kernel normalization in regression Is it a good idea to normalize your kernel in the Gaussian Process Regression framework?
By normalization, I mean $K_{norm}(x,y) = \frac{K(x,y)}{\sqrt{K(x,x)\times K(y,y)}}$.
For instance, take a polynomial kernel $K(x,y) = (<x,y>+b)^{d}$, where $d >0$. Thus, the "standard" polynomial kernel is not normalized. Is there any advantage of normalizing it?
 A: You are correct to note that certain kernels (polynomial being the prime example) suffect from numerical instability. This is pronounced as if $\langle x,y\rangle +b < 0$ then $(\langle x,y\rangle+b)^d \rightarrow 0$ as $d$ increases and vice versa if $\langle x,y\rangle +b > 0$ then $(\langle x,y\rangle+b)^d \rightarrow \infty$ as $d$ increases.
In general, it is more relevant to try to normalise the inputs $x$ and $y$ beforehand. For example, in R the code poly creates by default orthogonal polynomial to directly account for the inherit numerical stability of raising small or large number to increases powers. As we can cannot directly do that for a GPR the next best thing is to normalise our input.
To that extent and quoting directly from the Rasmussen & Williams (2007), Gaussian Processes for Machine Learning, Chapt. 4:
"For regression problems the polynomial kernel is a rather strange choice as the prior variance grows rapidly with $|x|$ for $|x|>1$. However, such kernels have proved effective in high-dimensional classification problems (e.g. take to be a vectorized binary image) where the input data are binary or grey scale normalized to [−1,1] on each dimension [Schoelkopf and Smola, 2002, sec. 7.8]".
So to bring everything together: Focus normalising the inputs rather than the kernel (or use a kernel that allows accounting for the scale of the inputs used (e.g. Matérn, RBF, etc.)).
