3
$\begingroup$

I am studying "Pattern Recognition and Machine Learning" by Christopher Bishop and I'm trying to understand his solution in the solution manual to exercise 1.4.

The problem statement for exercise 1.4 of the textbook is as follows:

Consider a probability density $p_x(x)$ defined over a continuous variable $x$, and suppose that we make a nonlinear change of variable using $x = g(y)$, so that the density transforms according to (1.27). By differentiating (1.27), show that the location $\tilde{y}$ of the maximum of the density in $y$ is not in general related to the location $\tilde{x}$ of the maximum of the density over $x$ by the simple functional relation $\tilde{x} = g(\tilde{y})$ as a consequence of the Jacobian factor. This shows that the maximum of a probability density (in contrast to a simple function) is dependent on the choice of variable. Verify that, in the case of a linear transformation, the location of the maximum transforms in the same way as the variable itself.

Equation 1.27 referenced above is

$$\begin{align} p_y(y) &= p_x(x) \left| \dfrac{dx}{dy} \right| \\ &= p_x(g(y)) |g'(y)| \tag{1.27} \end{align}$$

The following is the solution from the solutions manual:

We are often interested in finding the most probable value for some quantity. In the case of probability distributions over discrete variables this poses little problem. However, for continuous variables there is a subtlety arising from the nature of probability densities and the way they transform under non-linear changes of variable.

Consider first the way a function $f(x)$ behaves when we change to a new variable $y$ where the two variables are related by $x = g(y)$. This defines a new function of $y$ given by

$$\tilde{f}(y) = f(g(y)) \tag{2}$$􏰚

Suppose $f(x)$ has a mode (i.e. a maximum) at $\hat{x}$ so that $f'(\hat{x}) = 0$. The corresponding mode of $\tilde{f}(y)$ will occur for a value $\hat{y}$ obtained by differentiating both sides of (2) with respect to $y$

$$\tilde{f} \ ' (\hat{y}) = f'(g(\hat{y})) g'(\hat{y}) = 0 \tag{3}$$

Assuming $g'(\hat{y}) \not= 0$ at the mode, then $f'(g(\hat{y})) = 0$. However, we know that $f'(\hat{x}) = 0$, and so we see that the locations of the mode expressed in terms of each of the variables $x$ and $y$ are related by $\hat{x} = g(\hat{y})$, as one would expect. Thus, finding a mode with respect to the variable $x$ is completely equivalent to first transforming to the variable $y$, then finding a mode with respect to $y$, and then transforming back to $x$.

Now consider the behaviour of a probability density $p_x(x)$ under the change of variables $x = g(y)$, where the density with respect to the new variable is $p_y(y)$ and is given by ((1.27)). Let us write $g'(y) = s|g'(y)|$ where $s \in \{-1, +1\}$. Then ((1.27)) can be written

$$p_y(y) = p_x(g(y))sg'(y). \tag{*}$$ Question 1: How do they get that $|g'(y)|=sg'(y)$ in (*) if $g'(y)=s|g(y)|$?

Differentiating both sides with respect to $y$ then gives

$$p_y'(y) = sp_x'(g(y))\{g'(y)\}^2 + sp_x(g(y))g''(y). \tag{4}$$

Due to the presence of the second term on the right hand side of (4) the relationship $\hat{x} = g(\hat{y})$ no longer holds. Question 2: I don't completely understand why doesn't the relationship > > hold anymore?

Thus the value of $x$ obtained by maximizing $p_x(x)$ will not be the value obtained by transforming to $p_y(y)$ then maximizing with respect to $y$ and then transforming back to $x$. This causes modes of densities to be dependent on the choice of variables. In the case of linear transformation, the second term on the right hand side of (4) vanishes, and so the location of the maximum transforms according to $\hat{x} = g(\hat{y})$.

This effect can be illustrated with a simple example, as shown in Figure 1. We begin by considering a Gaussian distribution $p_x(x)$ over $x$ with mean $\mu = 6$ and standard deviation $\sigma = 1$, shown by the red curve in Figure 1. Next we draw a sample of $N = 50,000$ points from this distribution and plot a histogram of their values, which as expected agrees with the distribution $p_x(x)$.

enter image description here

Now consider a non-linear change of variables from $x$ to $y$ gives by

$$x = g(y) = \ln(y) - \ln(1 - y) + 5 \tag{5}$$

The inverse of this function is given by

$$y = g^{-1}(x) = \dfrac{1}{1 + \exp(-x + 5)} \ \tag{6}$$

which is a logistic sigmoid function, and is shown in Figure 1 by the blue curve. If we simply transform $p_x(x)$ as a function of $x$ we obtain the green curve $p_x(g(y))$ shown in Figure 1, and we see that the mode of the density $p_x(x)$ is transformed via the sigmoid function to the mode of this curve. However, the density over $y$ transforms instead according to (1.27) and is shown by the magenta curve on the left side of the diagram. Note that this has its mode shifted relative to the mode of the green curve.

To confirm this result we take out sample of 50,000 values of $x$, evaluate the corresponding values of $y$ using (6), and then plot a histogram of their values. We see that this histogram matches the magenta curve in Figure 1 and not the green curve!

If somebody could answer the 2 questions I asked that would help a lot in understanding the solution.

$\endgroup$

1 Answer 1

1
$\begingroup$

Question 1: How do they get that |𝑔′(𝑦)|=𝑠𝑔′(𝑦) in (*) if 𝑔′(𝑦)=𝑠|𝑔(𝑦)|?

Since $s\in \{1,-1\}$, we have $1/s=s$, therefore $s$ can be on either side of the equation.

Question 2: I don't completely understand why doesn't the relationship hold anymore?

If the second term didn't exist, we'd solve the following to find $\hat y$, i.e. the new mode: $$p_y'(\hat y) = sp_x'(g(\hat y))\{g'(\hat y)\}^2=0$$

Assuming the derivative term is non-zero, like the previous example, we'd have $p'_x(g(\hat y))=0$, which would at least be satisfied by $\hat x= g(\hat y)$. However, due to the second term in the sum, we can't have this for sure.

$\endgroup$

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.