2
$\begingroup$

I read in this tutorial on page 20 that $KL$ divergence is invariant to affine transformation, but I think it is incorrect.

Say we have two 1D normal distributions $P_{1}(x) = \mathcal N(\mu_{1}, \sigma_{1})$ and $P_{2}(x) = \mathcal N(\mu_{2}, \sigma_{2})$. So that $$KL(P_1(x)\|P_{2}(x))= E_{1}(\ln \frac{P_{1}(x)}{P_{2}(x)}) = \ln(\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma_1^2+(\mu_1-\mu_2)^2)-\frac{1}{2}$$

If we define an affine transformation as $$x^{'} = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$$

We will have $$P_1(x^{'}) = \sigma P_1(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ and $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$ Then, the $KL$ divergence for the two transformed distributions is $$KL(P_1(x')\|P_2(x')) = E'_1(\ln \frac{P_1(x')}{P_2(x')}) = \ln (\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma^2 \sigma_1^2+(\mu_1-\mu_2)^2)-\frac{\sigma^2}{2}$$

So clearly, for such a simple case $KL$ divergence is not invariant.

However, $KL$ divergence is invariant under affine transformation is crucial for the proof in the tutorial that I referred to.

So, have I misunderstood something?

EDIT:

I think part of my misunderstanding lies in the way that I calculate $P_1(x')$ and $P_2(x')$. So I will expand this part so others can see where I got it wrong. $$P_1(x') = \sigma P_1(x) = \sigma P_1(\mu_1+\sigma (x'-\mu_1))$$ given that $$P_1(x)=\mathcal N(\mu_1, \sigma_1)$$ so, $$\sigma P_1(\mu_1+\sigma (x'-\mu_1)) = \sigma \frac{1}{\sqrt{2\pi}\sigma_1} e^{-\frac{1}{2\sigma_1^2}(\sigma (x' - \mu_1))^2} = \frac{1}{\sqrt{2\pi} \frac{\sigma_1}{\sigma}} e^{-\frac{1}{2\frac{\sigma_1^2}{\sigma^2}}((x' - \mu_1))^2} = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ Then in the exact the same way, I have $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$

Is there any problem with this?

$\endgroup$
3
$\begingroup$

There are a few mistakes in your math. For example, when you expand the expectation, it seems you dropped the integral and also the $P_1(x)$ term.

Write $y(x) = mx + c$. Recall that $P(x) dx = P(y) dy$. This is easy to see since $dy/dx = m$ and it makes sense that $P(x) = mP(y)$.

Then we can go through with this proof from wikipedia which shows KL is invariant: enter image description here

$\endgroup$
  • $\begingroup$ I cannot see what is wrong with the substitution that I did. From $P_2(x')dx' = P_2(x)dx $, I have $P_2(x') = P_2(x)\sigma$ which is exactly what you pointed out. However, $P_2(x')$ is dependent of $x'$ but not $x$, so I substitute $x$ with $x'$ by $x = \mu_1+\sigma(x'-\mu_1)$, so $P_2(x')$ will be that normal distribution. What is wrong with this substitution? $\endgroup$ – meTchaikovsky Apr 21 '18 at 22:31
  • $\begingroup$ You can't have it that $P_2(x')$ depends on $x'$ but not $x$ and also that $x$ is an affine transformation of $x'$. $\endgroup$ – jbowman Apr 22 '18 at 1:19
  • $\begingroup$ @ jbowman sorry, I haven't made myself clear, what I meant is that I have $P_2(x') = P_2(x)\frac{dx}{dx'} = \sigma P_2(x)$. Since we have the inverse of the affine transformation $x = \mu_1 + \sigma (x' - \mu_1)$, then I substitute the $x$ in $\sigma P_2(x)$ by $x'$, thus I have $P_2(x')$ that is represented by $x'$. $\endgroup$ – meTchaikovsky Apr 22 '18 at 1:31
  • $\begingroup$ @shimao I found my mistake, I forget that I'm integrating over a new distribution so that I mistook $\int dx' P_1(x')(x'-\mu_1)^2 = \sigma_1^2$, which should be $\frac{\sigma_1^2}{m^2}$ $\endgroup$ – meTchaikovsky Apr 22 '18 at 1:45
  • $\begingroup$ @shimao from the proof, I think KL divergence is invariant to transformations which have an inverse which is differentiable, is it correct? $\endgroup$ – meTchaikovsky Apr 22 '18 at 3:12
0
$\begingroup$

I made a serious mistake while calculating the $KL$ divergence between the two 1D normal distributions. It is this mistake that causes me to doubt whether $KL$ divergence is invariant to affine transformation.

Where did I make the mistake:

When evaluating the expected value of $$(x' - \mu_1)^2$$ over the distribution $P_1(x')$, I made the mistake $$\int dx'P_1(x')(x'-\mu_1)^2 = \sigma_1^2$$ However, $P_1(x') = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$, so $$\int dx'P_1(x')(x'-\mu_1)^2 = \frac{\sigma_1^2}{\sigma^2}$$ By making this correction, we will have $$KL(P_1(x')\|P_2(x')) = KL(P_1(x)\|P_2(x))$$ which means KL divergence is invariant to the affine transformation $x' = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.