From "In All Likelihood: Statistical Modeling and Inference Using Likelihood" by Y. Pawitan, the likelihood of a re-parameterization $\theta\mapsto g(\theta)=\psi$ is defined as $$ L^*(\psi)=\max_{\{\theta:g(\theta)=\psi\}} L(\theta) $$ so that if $g$ is one-to-one, then $L^*(\psi)=L(g^{-1}(\psi))$ (p. 45). I am trying to show Exercise 2.20 which states that if $\theta$ is scalar (and I presume that $g$ is supposed to be a scalar function as well), then $$ I^*(g(\hat{\theta}))=I(\hat{\theta})\left|\frac{\partial g(\hat{\theta})}{\partial \hat{\theta}}\right|^{-2}, $$ where $$ I(\theta)=-\frac{\partial^2}{\partial\theta^2} l(\theta) $$ is the observed Fisher information and $l(\theta)=\log L(\theta)$.

If $g$ is one-to-one then this is straightforward using the chain-rule and the invariance principle. I am just wondering about a few things:

  1. Why does he insist on writing the absolute value? This could be left out, right?
  2. By $\frac{\partial g(\hat{\theta})}{\partial \hat{\theta}}$ he means the function $\frac{\partial g(\theta)}{\partial \theta}$ evaluated at $\theta=\hat{\theta}$, right? If this is the case, then isn't it a poor choice of notation? I believe that the usual shorthand notation for this woruld be $\frac{\partial g(\hat{\theta})}{\partial \theta}$.
  3. How is this shown when $g$ is not necessarily one-to-one?
  • $\begingroup$ The absolute value may stem from the formula that is used to transform a probability density. But for the score you, indeed, do not use the absolute values. It is not only superfluous, it is also wrong (it only doesn't matter because there is a square in this way of expressing the Fisher information). $\endgroup$ Nov 14 '20 at 14:55
  1. The absolute value is unnecessary. It may be just a typo.

  2. You're right. An even better notation would be $\frac{dg(\theta)}{d\theta}\Bigg|_{\theta=\hat{\theta}}$.

  3. It doesn't hold in general. Fix some $\psi_0$ and define $g:\mathbb{R}\to\mathbb{R}$ by $g(\theta)=\psi_0$. The rhs would be undefined since the derivative is zero for every $\theta$.

A sketch of the regular case:

For smooth one-to-one $g$ with $\psi=g(\theta)$. Since, $d/d\psi = d\theta/d\psi\cdot d/d\theta$, we have $$ \begin{align} I^*(\psi) &= -\frac{d^2L^*(\psi)}{d\psi^2} = -\frac{d}{d\psi}\left(\frac{dL^*(\psi)}{d\psi} \right) = -\frac{d}{d\psi}\left(\frac{dL^*(\psi)}{d\theta} \frac{d\theta}{d\psi}\right) \\ &= - \frac{d^2L^*(\psi)}{d\theta^2}\left(\frac{d\theta}{d\psi}\right)^2 - \frac{dL^*(\psi)}{d\theta}\frac{d^2\theta}{d\psi^2} \frac{d\theta}{d\psi}\, . \end{align} $$ Therefore, $$ \begin{align} I^*(g(\hat{\theta})) &= -\frac{d^2L^*(g(\hat{\theta}))}{d\theta^2}\left(\frac{d\theta}{d\psi}\right)^2 - \frac{dL^*(g(\hat{\theta}))}{d\theta}\frac{d^2\theta}{d\psi^2} \frac{d\theta}{d\psi} \\ &= -\frac{d^2L(g^{-1}(g(\hat{\theta})))}{d\theta^2} \left(\frac{dg(\theta)}{d\theta}\Bigg|_{\theta=g^{-1}(g(\hat{\theta}))}\right)^{-2} - \frac{dL(g^{-1}(g(\hat{\theta})))}{d\theta}\frac{d^2\theta}{d\psi^2} \frac{d\theta}{d\psi} \\ &= I(\hat{\theta}) \left(\frac{dg(\theta)}{d\theta}\Bigg|_{\theta=\hat{\theta}}\right)^{-2} \, , \end{align} $$ in which we used $dL(g^{-1}(g(\hat{\theta})))/d\theta=dL(\hat{\theta})/d\theta=0$.

  • 1
    $\begingroup$ Thank you for addressing all my doubts and for that simple counter-example with constant $g$. Your sketch of the regular case is similar to what I've done, so it's all good. Thanks. $\endgroup$ Oct 4 '14 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.