Another post has addressed the fact that KL divergence is defined between a uniform distribution and a Gaussian distribution $$D_{\text{KL}}(\mathcal{U}(x) \parallel \mathcal{N}(x \mid \mu, \Sigma)) = \int \mathcal{U}(x) \log\frac{\mathcal{U}(x)}{\mathcal{N}(x \mid \mu, \Sigma)} dx$$

essentially because the Gaussian in the denominator has infinite support, and so will have non-zero density over the support of the uniform distribution - we don't have any numerical problems in theory. This post provides a closed-form solution for computing the KL divergence between two multivariate Gaussians. I am hoping to find a similar solution for the case I describe above, between a uniform and a multivariate Gaussian.

Does a closed-form solution exist for arbitrary dimensions? If not that, what about in the 2-D case? If none of the above, how would I go about estimating this quantity aside from a pure sampling-based approximation?


1 Answer 1


It depends what the support of the uniform distribution looks like. But if you assume that it is supported on an axis-aligned rectangle $[a,b]\times [c,d]$ then it works out simply. Letting $u={\frac 1 {(b-a)(d-c)}}$, we have

\begin{eqnarray*} &&\int_{[a,b]\times [c,d]} u\left(\log(u)+{\frac 1 2}(x-\mu)^tC^{-1}(x-\mu)+{\frac 1 2}\log |C|+{\frac {1} {2}}\log 2\pi\right)dx\\ & = &\log(u)+{\frac 1 2}\log |C|+{\frac {1} {2}}\log 2\pi+{\frac 1 2}u\int_{[a,b]\times [c,d]}(x-\mu)^tC^{-1}(x-\mu)dx \end{eqnarray*}

Now, for simplicitly I'll take $\mu=0$ (although this is actually no loss of generality, because you can compensate for this by translating the bounds of the rectangle). Let $C^{-1}_{ij}$ denote the entries of $C^{-1}$. Tye integral is

\begin{eqnarray*} &&\int_{[a,b]\times [c,d]}x_1^2C^{-1}_{11}+2x_1x_2C^{-1}_{12}+x_2^2C^{-1}_{22}dx_1dx_2\\ & = & \int_{c}^d {\frac {x_1^3C^{-1}_{11}}{3}}+x_1^2x_2C^{-1}_{12}|_{x_1=a}^b +(b-a)x_2^2C^{-1}_{22}dx_2\\ & = &\int_{c}^d {\frac {(b^3-a^3)C^{-1}_{11}}{3}}+(b^2-a^2)x_2C^{-1}_{12} +(b-a)x_2^2C^{-1}_{22}dx_2\\ & = & {\frac {(d-c)(b^3-a^3)C^{-1}_{11}}{3}}+{\frac 1 2}(d^2-c^2)(b^2-a^2)C^{-1}_{12}+{\frac {(b-a)(d^3-c^3)C^{-1}_{22}}{3}} \end{eqnarray*}

In higher dimensions, you have to evaluate integrals like $\int_{\prod_i [a_i,b_i]} x_px_qC^{-1}_{ab} \prod_i dx_i$. In case $p=q$, the integral is $\prod_{i\neq p}(b_i-a_i)C^{-1}_{pp}(b_p^3-a_p^3)/3$, while if $p\neq q$ it is $\prod_{i\neq p,q}(b_i-a_i)C^{-1}_{pq}(b_p^2-a_p^2)(b_q^2-a_q^2)/4$. So in arbtirary dimensions you get the formula

$$\int_{\prod_i [a_i,b_i]}x^tC^{-1}xdx=\sum_p\prod_{i\neq p}(b_i-a_i)C^{-1}_{pp}(b_p^3-a_p^3)/3+ \sum_{p\neq q}\prod_{i\neq p,q}(b_i-a_i)C^{-1}_{pq}(b_p^2-a_p^2)(b_q^2-a_q^2)/4$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.