I read in this tutorial on page 20 that $KL$ divergence is invariant to affine transformation, but I think it is incorrect.

Say we have two 1D normal distributions $P_{1}(x) = \mathcal N(\mu_{1}, \sigma_{1})$ and $P_{2}(x) = \mathcal N(\mu_{2}, \sigma_{2})$. So that $$KL(P_1(x)\|P_{2}(x))= E_{1}(\ln \frac{P_{1}(x)}{P_{2}(x)}) = \ln(\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma_1^2+(\mu_1-\mu_2)^2)-\frac{1}{2}$$

If we define an affine transformation as $$x^{'} = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$$

We will have $$P_1(x^{'}) = \sigma P_1(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ and $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$ Then, the $KL$ divergence for the two transformed distributions is $$KL(P_1(x')\|P_2(x')) = E'_1(\ln \frac{P_1(x')}{P_2(x')}) = \ln (\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma^2 \sigma_1^2+(\mu_1-\mu_2)^2)-\frac{\sigma^2}{2}$$

So clearly, for such a simple case $KL$ divergence is not invariant.

However, $KL$ divergence is invariant under affine transformation is crucial for the proof in the tutorial that I referred to.

So, have I misunderstood something?

EDIT:

I think part of my misunderstanding lies in the way that I calculate $P_1(x')$ and $P_2(x')$. So I will expand this part so others can see where I got it wrong. $$P_1(x') = \sigma P_1(x) = \sigma P_1(\mu_1+\sigma (x'-\mu_1))$$ given that $$P_1(x)=\mathcal N(\mu_1, \sigma_1)$$ so, $$\sigma P_1(\mu_1+\sigma (x'-\mu_1)) = \sigma \frac{1}{\sqrt{2\pi}\sigma_1} e^{-\frac{1}{2\sigma_1^2}(\sigma (x' - \mu_1))^2} = \frac{1}{\sqrt{2\pi} \frac{\sigma_1}{\sigma}} e^{-\frac{1}{2\frac{\sigma_1^2}{\sigma^2}}((x' - \mu_1))^2} = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ Then in the exact the same way, I have $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$

Is there any problem with this?

up vote 3 down vote accepted

There are a few mistakes in your math. For example, when you expand the expectation, it seems you dropped the integral and also the $P_1(x)$ term.

Write $y(x) = mx + c$. Recall that $P(x) dx = P(y) dy$. This is easy to see since $dy/dx = m$ and it makes sense that $P(x) = mP(y)$.

Then we can go through with this proof from wikipedia which shows KL is invariant: enter image description here

  • I cannot see what is wrong with the substitution that I did. From $P_2(x')dx' = P_2(x)dx $, I have $P_2(x') = P_2(x)\sigma$ which is exactly what you pointed out. However, $P_2(x')$ is dependent of $x'$ but not $x$, so I substitute $x$ with $x'$ by $x = \mu_1+\sigma(x'-\mu_1)$, so $P_2(x')$ will be that normal distribution. What is wrong with this substitution? – meTchaikovsky Apr 21 at 22:31
  • You can't have it that $P_2(x')$ depends on $x'$ but not $x$ and also that $x$ is an affine transformation of $x'$. – jbowman Apr 22 at 1:19
  • @ jbowman sorry, I haven't made myself clear, what I meant is that I have $P_2(x') = P_2(x)\frac{dx}{dx'} = \sigma P_2(x)$. Since we have the inverse of the affine transformation $x = \mu_1 + \sigma (x' - \mu_1)$, then I substitute the $x$ in $\sigma P_2(x)$ by $x'$, thus I have $P_2(x')$ that is represented by $x'$. – meTchaikovsky Apr 22 at 1:31
  • @shimao I found my mistake, I forget that I'm integrating over a new distribution so that I mistook $\int dx' P_1(x')(x'-\mu_1)^2 = \sigma_1^2$, which should be $\frac{\sigma_1^2}{m^2}$ – meTchaikovsky Apr 22 at 1:45
  • @shimao from the proof, I think KL divergence is invariant to transformations which have an inverse which is differentiable, is it correct? – meTchaikovsky Apr 22 at 3:12

I made a serious mistake while calculating the $KL$ divergence between the two 1D normal distributions. It is this mistake that causes me to doubt whether $KL$ divergence is invariant to affine transformation.

Where did I make the mistake:

When evaluating the expected value of $$(x' - \mu_1)^2$$ over the distribution $P_1(x')$, I made the mistake $$\int dx'P_1(x')(x'-\mu_1)^2 = \sigma_1^2$$ However, $P_1(x') = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$, so $$\int dx'P_1(x')(x'-\mu_1)^2 = \frac{\sigma_1^2}{\sigma^2}$$ By making this correction, we will have $$KL(P_1(x')\|P_2(x')) = KL(P_1(x)\|P_2(x))$$ which means KL divergence is invariant to the affine transformation $x' = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.