# KL divergence invariant to affine transformation?

I read in this tutorial on page 20 that $KL$ divergence is invariant to affine transformation, but I think it is incorrect.

Say we have two 1D normal distributions $P_{1}(x) = \mathcal N(\mu_{1}, \sigma_{1})$ and $P_{2}(x) = \mathcal N(\mu_{2}, \sigma_{2})$. So that $$KL(P_1(x)\|P_{2}(x))= E_{1}(\ln \frac{P_{1}(x)}{P_{2}(x)}) = \ln(\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma_1^2+(\mu_1-\mu_2)^2)-\frac{1}{2}$$

If we define an affine transformation as $$x^{'} = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$$

We will have $$P_1(x^{'}) = \sigma P_1(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ and $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$ Then, the $KL$ divergence for the two transformed distributions is $$KL(P_1(x')\|P_2(x')) = E'_1(\ln \frac{P_1(x')}{P_2(x')}) = \ln (\frac{\sigma_{2}}{\sigma_{1}}) + \frac{1}{2\sigma_2^2}(\sigma^2 \sigma_1^2+(\mu_1-\mu_2)^2)-\frac{\sigma^2}{2}$$

So clearly, for such a simple case $KL$ divergence is not invariant.

However, $KL$ divergence is invariant under affine transformation is crucial for the proof in the tutorial that I referred to.

So, have I misunderstood something?

EDIT:

I think part of my misunderstanding lies in the way that I calculate $P_1(x')$ and $P_2(x')$. So I will expand this part so others can see where I got it wrong. $$P_1(x') = \sigma P_1(x) = \sigma P_1(\mu_1+\sigma (x'-\mu_1))$$ given that $$P_1(x)=\mathcal N(\mu_1, \sigma_1)$$ so, $$\sigma P_1(\mu_1+\sigma (x'-\mu_1)) = \sigma \frac{1}{\sqrt{2\pi}\sigma_1} e^{-\frac{1}{2\sigma_1^2}(\sigma (x' - \mu_1))^2} = \frac{1}{\sqrt{2\pi} \frac{\sigma_1}{\sigma}} e^{-\frac{1}{2\frac{\sigma_1^2}{\sigma^2}}((x' - \mu_1))^2} = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$$ Then in the exact the same way, I have $$P_2(x^{'}) = \sigma P_2(x = \mu_1+ \sigma(x' - \mu_1)) = \mathcal N(\mu_1-\frac{1}{\sigma}(\mu_1-\mu_2), \frac{\sigma_2^2}{\sigma^2})$$

Is there any problem with this?

There are a few mistakes in your math. For example, when you expand the expectation, it seems you dropped the integral and also the $P_1(x)$ term.

Write $y(x) = mx + c$. Recall that $P(x) dx = P(y) dy$. This is easy to see since $dy/dx = m$ and it makes sense that $P(x) = mP(y)$.

Then we can go through with this proof from wikipedia which shows KL is invariant:

• I cannot see what is wrong with the substitution that I did. From $P_2(x')dx' = P_2(x)dx$, I have $P_2(x') = P_2(x)\sigma$ which is exactly what you pointed out. However, $P_2(x')$ is dependent of $x'$ but not $x$, so I substitute $x$ with $x'$ by $x = \mu_1+\sigma(x'-\mu_1)$, so $P_2(x')$ will be that normal distribution. What is wrong with this substitution? – meTchaikovsky Apr 21 '18 at 22:31
• You can't have it that $P_2(x')$ depends on $x'$ but not $x$ and also that $x$ is an affine transformation of $x'$. – jbowman Apr 22 '18 at 1:19
• @ jbowman sorry, I haven't made myself clear, what I meant is that I have $P_2(x') = P_2(x)\frac{dx}{dx'} = \sigma P_2(x)$. Since we have the inverse of the affine transformation $x = \mu_1 + \sigma (x' - \mu_1)$, then I substitute the $x$ in $\sigma P_2(x)$ by $x'$, thus I have $P_2(x')$ that is represented by $x'$. – meTchaikovsky Apr 22 '18 at 1:31
• @shimao I found my mistake, I forget that I'm integrating over a new distribution so that I mistook $\int dx' P_1(x')(x'-\mu_1)^2 = \sigma_1^2$, which should be $\frac{\sigma_1^2}{m^2}$ – meTchaikovsky Apr 22 '18 at 1:45
• @shimao from the proof, I think KL divergence is invariant to transformations which have an inverse which is differentiable, is it correct? – meTchaikovsky Apr 22 '18 at 3:12

I made a serious mistake while calculating the $KL$ divergence between the two 1D normal distributions. It is this mistake that causes me to doubt whether $KL$ divergence is invariant to affine transformation.

Where did I make the mistake:

When evaluating the expected value of $$(x' - \mu_1)^2$$ over the distribution $P_1(x')$, I made the mistake $$\int dx'P_1(x')(x'-\mu_1)^2 = \sigma_1^2$$ However, $P_1(x') = \mathcal N(\mu_1, \frac{\sigma_1^2}{\sigma^2})$, so $$\int dx'P_1(x')(x'-\mu_1)^2 = \frac{\sigma_1^2}{\sigma^2}$$ By making this correction, we will have $$KL(P_1(x')\|P_2(x')) = KL(P_1(x)\|P_2(x))$$ which means KL divergence is invariant to the affine transformation $x' = \mu_1 + \frac{1}{\sigma}(x - \mu_1)$.