Computing KL divergence between uniform and multivariate Gaussian

Question

Another post has addressed the fact that KL divergence is defined between a uniform distribution and a Gaussian distribution $$D_{\text{KL}}(\mathcal{U}(x) \parallel \mathcal{N}(x \mid \mu, \Sigma)) = \int \mathcal{U}(x) \log\frac{\mathcal{U}(x)}{\mathcal{N}(x \mid \mu, \Sigma)} dx$$

essentially because the Gaussian in the denominator has infinite support, and so will have non-zero density over the support of the uniform distribution - we don't have any numerical problems in theory. This post provides a closed-form solution for computing the KL divergence between two multivariate Gaussians. I am hoping to find a similar solution for the case I describe above, between a uniform and a multivariate Gaussian.

Does a closed-form solution exist for arbitrary dimensions? If not that, what about in the 2-D case? If none of the above, how would I go about estimating this quantity aside from a pure sampling-based approximation?

Answer 1

It depends what the support of the uniform distribution looks like. But if you assume that it is supported on an axis-aligned rectangle $[a,b]\times [c,d]$ then it works out simply. Letting $u={\frac 1 {(b-a)(d-c)}}$, we have

\begin{eqnarray*} &&\int_{[a,b]\times [c,d]} u\left(\log(u)+{\frac 1 2}(x-\mu)^tC^{-1}(x-\mu)+{\frac 1 2}\log |C|+{\frac {1} {2}}\log 2\pi\right)dx\\ & = &\log(u)+{\frac 1 2}\log |C|+{\frac {1} {2}}\log 2\pi+{\frac 1 2}u\int_{[a,b]\times [c,d]}(x-\mu)^tC^{-1}(x-\mu)dx \end{eqnarray*}

Now, for simplicitly I'll take $\mu=0$ (although this is actually no loss of generality, because you can compensate for this by translating the bounds of the rectangle). Let $C^{-1}_{ij}$ denote the entries of $C^{-1}$. Tye integral is

\begin{eqnarray*} &&\int_{[a,b]\times [c,d]}x_1^2C^{-1}_{11}+2x_1x_2C^{-1}_{12}+x_2^2C^{-1}_{22}dx_1dx_2\\ & = & \int_{c}^d {\frac {x_1^3C^{-1}_{11}}{3}}+x_1^2x_2C^{-1}_{12}|_{x_1=a}^b +(b-a)x_2^2C^{-1}_{22}dx_2\\ & = &\int_{c}^d {\frac {(b^3-a^3)C^{-1}_{11}}{3}}+(b^2-a^2)x_2C^{-1}_{12} +(b-a)x_2^2C^{-1}_{22}dx_2\\ & = & {\frac {(d-c)(b^3-a^3)C^{-1}_{11}}{3}}+{\frac 1 2}(d^2-c^2)(b^2-a^2)C^{-1}_{12}+{\frac {(b-a)(d^3-c^3)C^{-1}_{22}}{3}} \end{eqnarray*}

In higher dimensions, you have to evaluate integrals like $\int_{\prod_i [a_i,b_i]} x_px_qC^{-1}_{ab} \prod_i dx_i$. In case $p=q$, the integral is $\prod_{i\neq p}(b_i-a_i)C^{-1}_{pp}(b_p^3-a_p^3)/3$, while if $p\neq q$ it is $\prod_{i\neq p,q}(b_i-a_i)C^{-1}_{pq}(b_p^2-a_p^2)(b_q^2-a_q^2)/4$. So in arbtirary dimensions you get the formula

$$\int_{\prod_i [a_i,b_i]}x^tC^{-1}xdx=\sum_p\prod_{i\neq p}(b_i-a_i)C^{-1}_{pp}(b_p^3-a_p^3)/3+ \sum_{p\neq q}\prod_{i\neq p,q}(b_i-a_i)C^{-1}_{pq}(b_p^2-a_p^2)(b_q^2-a_q^2)/4$$