Entropy: Given y=Ax, then H[y] = H[X] + ln |A|

Question

The entropy of H[y] is given by:

$$H[y] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$

Now, suppose that I want to make a linear transformation of vector $\mathbf{y}$ to change the variables of the integration to vector $\mathbf{x}$ according to:

$$\mathbf{y}=\mathbf{A}\mathbf{x}$$

Further, suppose that vector $\mathbf{x}$ is a continuous random variable with distribution $p_x(x)$ and corresponding entropy H[x].

What is H[y] in terms of H[x]?

Step 1:

$$H[\mathbf{y}] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$

Step 2: $$H[\mathbf{y}] = - \int \bigg(\frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg) \ln \bigg( \frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg)~ \bigg| \frac{\partial y}{\partial x} \bigg| d\mathbf{x}$$

Step 3: $$H[\mathbf{y}] = -\int p_x(\mathbf{x}) \ln \bigg(\frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg)d\mathbf{x}$$

Step 4: $$H[\mathbf{y}] = -\int p_x(\mathbf{x}) \ln p_x(\mathbf{x}) d\mathbf{x} - \int p_x(\mathbf{x}) \ln\bigg( \frac{1}{|\mathbf{A}|}\bigg)d\mathbf{x}$$

Step 5: $$H[\mathbf{y}] = H[\mathbf{x}] + ln |\mathbf{A}|$$

My question is as follows:

What happened between step 1 and step 2? I wanted to see the details of this step broken down so that i could follow it. The other steps I can follow without any problems...

Answer 1

Under a nonlinear change of variable, a probability density transforms differently from a simple function, due to the Jacobian factor.

For instance, if we consider a change of variables $x=g(y)$, then a function $f(x)$ becomes $\tilde{f}(x) = f(g(x))$.

Now consider a probability density $p_x(x)$ that corresponds to a density $p_y(y)$ with respect to a new variable y, where the suffixes denote the fact that $p_x(x)$ and $p_y(y)$ are different densities.

Observations falling in the range $(x, x + \delta x)$ will, for small values of $\delta x$, be transformed into the range $(y, y + \delta y)$ where:

$$p_x(x) \delta x = p_y(y) \delta y$$

and hence:

$$p_y(y) = p_x(x) \bigg| \frac{dx}{dy}\bigg|$$