# KL divergence between GPs undefined?

In Theorem 1 in Sun et al, Functional Variational Bayesian Neural Networks, 2019, the authors state the the KL divergence between stochastic processes in the supremum over KL divergence between vectors each element of which is sampled from the respective process, meaning

$$$$KL(f \| g) = \sup_{n \in N, x_1, …, x_n \in X} KL([ f(x_1).. f(x_n)]\ \|\ [g(x_1).. g(x_n) ]).$$$$

I constructed a simple example for which it doesn't make sense, any pointers on where it's wrong (or violates implicit assumptions in the paper) appreciated.

Take $$g \sim GP(0, k)$$, $$f \sim GP(0, 2k)$$, two zero-mean Gaussian processes; the kernel of g is two times the kernel of g. The kernel k can be arbitrary. The KL divergence between vectors is then the KL divergence between multivariate Gaussians, and takes the well-known form (remember that we assumed zero means)

$$$$KL([ f(x_1).. f(x_n)]\ \|\ [g(x_1).. g(x_n) ]) = \frac{1}{2} \left( \mathrm{Tr} \left[ \Sigma_2^{-1} \Sigma_1 \right] - \log \left[\det \Sigma_1 / \det \Sigma_2\right] - n \right)$$$$

for $$\Sigma_1$$ the Gram matrix of $$2k$$ evaluated on $$x_1, \dots, x_n$$, and $$\Sigma_2$$ the Gram matrix of $$k$$ evaluated on $$x_1, \dots, x_n$$. Then $$2\Sigma_2=\Sigma_1$$, and we can simplify further,

$$$$KL([ f(x_1).. f(x_n)]\ \|\ [g(x_1).. g(x_n) ]) = \frac{1}{2} \left( 2n - n \log 2 - n \right) = \frac{n(1- \log 2)}{2}.$$$$

The supremum of this is infinity. That would imply the KL between two GPs, with arbitrary kernels, is not defined, which seems unlikely! What's the issue?

• The KL here is infinite for the wrong reason. Why not consider a modified definition where you normalize by $n$ before taking the supremum ? Then the KL becomes a constant $c>0$. Commented Jul 2 at 13:31