4
$\begingroup$

I want to prove This formula:

enter image description here

The score function is basically the derivative of the maximum likelihood's log, so to get the information I make another derivative of that:

$$ -E[∂/∂θ s(X;θ)] = -E[∂/∂θ (∂log f(X;θ) / ∂θ)] = -E[∂^2/∂θ^2 log f(X;θ)] $$

Now I did $$ -E[∂^2/∂θ^2 log f(X;θ)] = -E[(∂^2/∂θ^2 f(X;θ)) / f(X;θ)] + E[(∂/∂θ f(X;θ)/f(X;θ))^2] $$

I did integral of $-E[(∂^2/∂θ^2 f(X;θ)) / f(X;θ)]$ which aventualy equals to zero, which leaves us with

$$ -E[∂^2/∂θ^2 log f(X;θ)] = 0 + E[(∂/∂θ f(X;θ)/f(X;θ))^2] $$

And since $$ E[(∂/∂θ f(X;θ)/f(X;θ))^2] = E[s(X;θ)^2] $$

This solves the proof.

What do you think?

$\endgroup$
3
  • $\begingroup$ $\mathsf I(\theta):=\mathbb E_\theta\left\{\left(\frac{\partial}{\partial\theta}\ln f_\theta(\mathbf X) \right)^2\right\} = -\mathbb E_\theta\left\{\frac{\partial^2}{\partial\theta^2}\ln f_\theta(\mathbf X) \right\}= -\mathbb E_\theta\left\{\frac{\partial}{\partial\theta}s_\theta(\mathbf X) \right\}.$ $\endgroup$ Commented Nov 12, 2022 at 13:49
  • $\begingroup$ Yeap this is exactly what I need to prove.. $\endgroup$ Commented Nov 12, 2022 at 14:06
  • $\begingroup$ On the Wikipedia page for the Fisher information there is a derivation for this. $\endgroup$ Commented Nov 12, 2022 at 15:29

1 Answer 1

0
$\begingroup$

Assuming the regularity conditions, let $\mathbf X\sim f_\theta(\mathbf x)$ w.r.t. a sigma-finite measure $\mu$ on $(\mathscr X,\mathfrak B(\mathscr X)).$

Now

\begin{align}0 &= \frac{\partial}{\partial \theta}\int_\mathscr X f_\theta(\mathbf x) ~\mathrm d\mu(\mathbf x) \\ &=\int_\mathscr X \frac{\partial}{\partial \theta}f_\theta(\mathbf x) ~\mathrm d\mu(\mathbf x)\\&=\int_\mathscr X \frac{\partial}{\partial\theta}\ln f_\theta(\mathbf x)\cdot f_\theta(\mathbf x) ~\mathrm d\mu(\mathbf x) \\\implies 0&=\int_\mathscr X\frac{\partial}{\partial\theta}\left[\frac{\partial}{\partial\theta}\ln f_\theta(\mathbf x)\cdot f_\theta(\mathbf x) \right]~\mathrm d\mu(\mathbf x)\\ 0&= \int_\mathscr X\frac{\partial^2}{\partial\theta^2}\ln f_\theta(\mathbf x)\cdot f_\theta(\mathbf x) ~\mathrm d\mu(\mathbf x)+ \int_\mathscr X\left\{\frac{\partial}{\partial\theta} \ln f_\theta(\mathbf x)\right\}^2 f_\theta(\mathbf x) ~\mathrm d\mu(\mathbf x)\\ &= \mathbb E_\theta\left\{\frac{\partial^2}{\partial\theta^2}\ln f_\theta(\mathbf X) \right\}+ \mathbb E_\theta\left\{\left(\frac{\partial}{\partial\theta}\ln f_\theta(\mathbf X) \right)^2\right\}.\tag 1\label 1\end{align}

Then, Fisher information

\begin{align}\mathsf I(\theta)&:=\mathbb E_\theta\left\{\left(\frac{\partial}{\partial\theta}\ln f_\theta(\mathbf X) \right)^2\right\} \\&\overset{\eqref{1}}{=} -\mathbb E_\theta\left\{\frac{\partial^2}{\partial\theta^2}\ln f_\theta(\mathbf X) \right\}\\&= -\mathbb E_\theta\left\{\frac{\partial}{\partial\theta}s_\theta(\mathbf X) \right\}.\end{align}

$\endgroup$
5
  • 3
    $\begingroup$ I wonder whether symbols like $\mathfrak B$ and $\mathscr X$ make this proof easy to read and follow. Also, do we need to use a measure theoretical approach? $\endgroup$ Commented Nov 12, 2022 at 15:43
  • $\begingroup$ I can get where your concern is coming from @SextusEmpiricus. Certainly it could be just done in the usual Riemannian integral. But my objective is to provide a clear exposition of the relevant derivation. In fact, the symbols are universal and for the sake of simplicity, I avoided the usage of null set in the integral not to be pedantic. The derivation is clear especially in its usage of the regularity conditions and the rest follows from the relation. But if you insist, I would alter those. $\endgroup$ Commented Nov 12, 2022 at 15:52
  • $\begingroup$ @User1865345 Thank you so much for the effort! What do you think about my proof? it might be a little bit messy I'll try to make it clearer. $\endgroup$ Commented Nov 12, 2022 at 16:13
  • $\begingroup$ @ProgrammingNoob it could be myself only but perhaps typesetting them properly could be more legible. $\endgroup$ Commented Nov 12, 2022 at 16:16
  • $\begingroup$ @User1865345 if these symbols would be universal, then why is it the first time I see them here? Your objective might be to provide a clear exposition of the relevant derivation, but in order to be 'clear' you need to provide a learning slope that is less steep. One can give a straightforward/direct proof of this equality for the Fisher information, but maybe people also want to be able to understand the proof and it should be described in a way that a wider public can follow. $\endgroup$ Commented Nov 12, 2022 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.