1
$\begingroup$

Bhattacharyya distance between two distributions $p$ and $q$ is defined as $D_B(p,q)=-\log(\int\sqrt{p(x)q(x)})dx$, The KL-divergence is defined as $D_{kl}(p||q)=\int p(x)\log(\frac{p(x)}{q(x)})dx$.

If $p$ and $q$ are Gaussian ( $p(x)=\mathcal{N}(x|\mu_p,\Sigma_p)$ and $q(x)=\mathcal{N}(x|\mu_q,\Sigma_q)$ ) then

$$ D_B(p,q) = \frac{1}{8}(\mu_p - \mu_q)^T(\frac{\Sigma_p+\Sigma_q}{2})^{-1}(\mu_p - \mu_q) + \frac{1}{2}\log(\frac{|\frac{\Sigma_p+\Sigma_q}{2}|}{|\Sigma_p|^{0.5}|\Sigma_q|^{0.5}}). $$ $$ D_{kl}(p||q)= \frac{1}{2}(\mu_p - \mu_q)^T(\Sigma_q)^{-1}(\mu_p - \mu_q) - k/2 + \frac{1}{2}trace(\Sigma_q^{-1}\Sigma_p) +\frac{1}{2}\log(\frac{|\Sigma_q|}{|\Sigma_p|}). $$

Now if $\mu_p=\begin{bmatrix} -0.6702 \\-1.1827 \end{bmatrix}$ , $\Sigma_p=10^{-5}\times\begin{bmatrix} 0.1966 & 0.0179\\ 0.0179 & 0.0024 \end{bmatrix}$ and $\mu_q=\begin{bmatrix} -0.6641 \\ -1.1803 \end{bmatrix}$ , $\Sigma_q=10^{-6}\times\begin{bmatrix} 0.6853 &0.3309 \\ 0.3309 &0.1618 \end{bmatrix}$ then:

$D_B(p,q) = 9.0679$, $D_{kl}(p||q)=176.3679$ and $D_{kl}(q||p)=237.6702$.

Now if we keep $\mu_p$ and $\mu_q$, but make $\Sigma_p$ and $\Sigma_q$ diagonal so $\mu_p=\begin{bmatrix} -0.6702 \\-1.1827 \end{bmatrix}$ , $\Sigma_p=10^{-5}\times\begin{bmatrix} 0.1966 & 0\\ 0 & 0.0024 \end{bmatrix}$ and $\mu_q=\begin{bmatrix} -0.6641 \\ -1.1803 \end{bmatrix}$ , $\Sigma_q=10^{-6}\times\begin{bmatrix} 0.6853 &0 \\ 0 &0.1618 \end{bmatrix}$, then:

$D_B(p,q) = 11.5250$, $D_{kl}(p||q)=45.8842$ and $D_{kl}(q||p)=131.5813$.

We can see that the Bhattacharyya distance increases when we make the covariances diagonal but KL divergence decreases. Because both Bhattacharyya distance and KL divergence measure dissimilarity I expected that both increase/decrease. Can anyone explain why this happens ?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.