1
$\begingroup$

The entropy of H[y] is given by:

$$H[y] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$

Now, suppose that I want to make a linear transformation of vector $\mathbf{y}$ to change the variables of the integration to vector $\mathbf{x}$ according to:

$$\mathbf{y}=\mathbf{A}\mathbf{x}$$

Further, suppose that vector $\mathbf{x}$ is a continuous random variable with distribution $p_x(x)$ and corresponding entropy H[x].

What is H[y] in terms of H[x]?

Step 1:

$$H[\mathbf{y}] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$

Step 2: $$H[\mathbf{y}] = - \int \bigg(\frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg) \ln \bigg( \frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg)~ \bigg| \frac{\partial y}{\partial x} \bigg| d\mathbf{x}$$

Step 3: $$H[\mathbf{y}] = -\int p_x(\mathbf{x}) \ln \bigg(\frac{p_x(\mathbf{x})}{|\mathbf{A}|}\bigg)d\mathbf{x}$$

Step 4: $$H[\mathbf{y}] = -\int p_x(\mathbf{x}) \ln p_x(\mathbf{x}) d\mathbf{x} - \int p_x(\mathbf{x}) \ln\bigg( \frac{1}{|\mathbf{A}|}\bigg)d\mathbf{x}$$

Step 5: $$H[\mathbf{y}] = H[\mathbf{x}] + ln |\mathbf{A}|$$

My question is as follows:

What happened between step 1 and step 2? I wanted to see the details of this step broken down so that i could follow it. The other steps I can follow without any problems...

$\endgroup$
1
$\begingroup$

Under a nonlinear change of variable, a probability density transforms differently from a simple function, due to the Jacobian factor.

For instance, if we consider a change of variables $x=g(y)$, then a function $f(x)$ becomes $\tilde{f}(x) = f(g(x))$.

Now consider a probability density $p_x(x)$ that corresponds to a density $p_y(y)$ with respect to a new variable y, where the suffixes denote the fact that $p_x(x)$ and $p_y(y)$ are different densities.

Observations falling in the range $(x, x + \delta x)$ will, for small values of $\delta x$, be transformed into the range $(y, y + \delta y)$ where:

$$p_x(x) \delta x = p_y(y) \delta y$$

and hence:

$$p_y(y) = p_x(x) \bigg| \frac{dx}{dy}\bigg|$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.