I was reviewing the section of Andrew Gelman's "Bayesian Data Analysis" on uninformative priors, and came across this explanation for why Jeffreys' prior is invariant to parameterization.

Bayesian Data Analysis, 3e p. 53

My question is simply how Gelman reasoned from the first line of the equation to the second.


So using the chain rule, is this the correct reasoning? If we take $\theta(\phi)$ as a function of $\phi$, then

$$ \frac{d^2\log p(y \mid \phi)}{d\phi^2} = \frac{d}{d\phi} \left( \frac{d \log p(y\mid\theta(\phi))}{d \theta} \frac{d\theta}{d\phi} \right) = \frac{d^2 \log p(y\mid\theta(\phi))}{d \theta^2} \left|\frac{d\theta}{d\phi} \right|^2 $$

  • 1
    $\begingroup$ en.wikipedia.org/wiki/Chain_rule $\endgroup$ – whuber Feb 11 '17 at 20:18
  • 2
    $\begingroup$ I agree with William Huber. It simply amounts to the chain rule of calculus. $\endgroup$ – Michael Chernick Feb 11 '17 at 20:36
  • $\begingroup$ Thanks both, I got myself confused because it was to the right of the conditional. Added an updated explanation in an edit. $\endgroup$ – ezhao15 Feb 11 '17 at 20:39
  • $\begingroup$ This explanation is of course restricted to the unidimensional case. $\endgroup$ – Xi'an Feb 12 '17 at 18:12

Regarding your edit, that's not right. You also need the product rule:

\begin{align*} \frac{d^2\log p(y | \phi)}{d\phi^2} &= \frac{d}{d\phi} \left( \frac{d \log p(y|\theta(\phi))}{d \theta} \frac{d\theta}{d\phi} \right) \tag{chain rule}\\ &= \left(\frac{d^2 \log p(y|\theta(\phi))}{d \theta d\phi}\right)\left( \frac{d\theta}{d\phi}\right) + \left(\frac{d \log p(y|\theta(\phi))}{d \theta}\right) \left( \frac{d^2\theta}{d\phi^2}\right) \tag{prod. rule} \\ &= \left(\frac{d^2 \log p(y|\theta(\phi))}{d \theta^2 }\right)\left( \frac{d\theta}{d\phi}\right)^2 + \left(\frac{d \log p(y|\theta(\phi))}{d \theta}\right) \left( \frac{d^2\theta}{d\phi^2}\right) \tag{chain rule} \end{align*} Then $\frac{d \log p(y|\theta(\phi))}{d \theta}$ (the "score function") is $0$ on average.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.