7
$\begingroup$

I've reed the following part of a sketch of the proof that the maximum likelihood estimator is asymptotically normal:

"Sketch of the second part of the proof. Recall that we may write the likelihood equation as $$\sum_{i=0}^n U_i(\hat\theta)=0$$ where $ U_i(\phi)$ denotes the derivative of $ \log\! f(Y_i;\phi) $ with respect to $\phi$. Let $ U_i'(\phi) $ denote the derivative of $ \log\!f(Y_i;\phi)$ with respect to $ \phi$ . Now a Taylor expansion around $ \phi=\theta$ yields:

$$\sum_{i=0}^n U_i(\phi )-\sum_{i=0}^n U_i(\theta)\approx \left( \sum_{i=0}^n U_i'(\theta) \right)(\phi-\theta)$$"

This Taylor expansion does not make any sense to me. I am familiar with a Taylor expansion of $f(x) $ at $a$ as: $$\sum_{i=0}^\infty\frac{f^{(n)}(a)}{n!}(x-a)^n$$ I can see that all terms except the second term from the Taylor expansion of the log likelihood are excluded but I do not recognise where this term comes from as the original term involved subtracting two summations.

$\endgroup$
  • 1
    $\begingroup$ Perhaps it would be clearer to characterize this as the definition of the derivative (together with the sum formula for derivatives). There's no need to invoke Taylor. $\endgroup$ – whuber Sep 28 '15 at 22:18
7
$\begingroup$

You should convince yourself that $f(x)-f(y)\approx f'(x)(x-y)$ is just another way to express Taylor expansion (under appropriate regularity assumptions).

Then, using linearity of the derivation, you can generalize it to the sum.

$\endgroup$
  • $\begingroup$ Is the taylor expansion of a sum of functions the same as the sum of the taylor expansions of the two separate functions? Then I might see it. $\endgroup$ – Joogs Sep 28 '15 at 21:07
  • $\begingroup$ @Joogs Yes, recall that a sum of differentiable functions is also differentiable. $\endgroup$ – JohnK Sep 28 '15 at 21:49
  • $\begingroup$ But is the taylor expansion of a sum of functions also the sum of the taylor expansions of the two functions if one is a function of x and the other a function of y? $\endgroup$ – Joogs Sep 28 '15 at 22:05
  • $\begingroup$ @Joogs, Yes! You just have to pay attention that you do not go too far away from the point (x,y) $\endgroup$ – RUser4512 Sep 29 '15 at 8:25
4
$\begingroup$

I think your problem is just the notation problem

Let us start from the beginning;

Suppose$X_1, X_2,...,X_n$ are i.i.d random variables with probability density function $f(x;\gamma)$

Usually, people will use $\theta$ here, I will keep it for later use.

The likelihood function is $L(\gamma;x)=f(x_1;\gamma)f(x_2;\gamma)...f(x_n;\gamma)$

The log likelihood function $l(\gamma;x)=log[f(x_1;\gamma)f(x_2;\gamma)...f(x_n;\gamma)]=\sum_{i=1}^nlogf(x_i;\gamma)$

As usual, we will take derivative of the log likelihood and set it to zero i.e $l'(\gamma;x)=0 $

or $\sum_{i=1}^nlog'f(x_i;\gamma)=0$

Here, you can see $U_i(\gamma)=log'f(x_i;\gamma)$

$\therefore l'(\gamma)=\sum_{i=1}^nU_i(\gamma)\tag{1}$ we omit $x$ here since the log likelihood function is a function of $\gamma $

Next we expand the function $l'(\gamma)$ into a Taylor series of order two abut $\theta$ .

$l'(\gamma)=l'(\theta)+\frac{l''(\theta)}{1!}(\gamma-\theta)^1+\frac{l'''(\theta)}{2!}(\gamma-\theta)^2$. This is the Taylor expanding for $l'(\gamma)$.

Next we evaluate the equation at $\phi$

$l'(\phi)=l'(\theta)+\frac{l''(\theta)}{1!}(\phi-\theta)^1+\frac{l'''(\theta)}{2!}(\phi-\theta)^2 \tag{2}$

Here you should see that $l'(\theta)=\sum_{i=1}^nU_i(\theta)$

and $l'(\phi)=\sum_{i=1}^nU_i(\phi)$

and $l''(\phi)=\sum_{i=1}^nU_i'(\phi)$

Ref (1)

If we ignore the third derivative term in (2) then your question will be answered here

We get that $\sum_{i=1}^n U_i(\phi )-\sum_{i=1}^n U_i(\theta)\approx \left( \sum_{i=1}^n U_i'(\theta) \right)(\phi-\theta)$

By the way I think $i$ usually start from $1$ not $0$ ,anyway it is just an index.

Let us don't stop here, we can go further to prove the theorem

We know that $l'(\phi)=0$

$\therefore l'(\theta)+\frac{l''(\theta)}{1!}(\phi-\theta)^1+\frac{l'''(\theta)}{2!}(\phi-\theta)^2=0$

i.e.

$l'(\theta)+(\phi-\theta)*[l''(\theta)+\frac{l'''(\theta)}{2}(\phi-\theta)]=0$

Next we rearrange the above terms:

$$(\phi-\theta)=\frac{l'(\theta)}{-l''(\theta)-\frac{l'''(\theta)}{2}(\phi-\theta)}$$

We multiply $\sqrt{n}$ for both side:

$$\sqrt{n}(\phi-\theta)=\frac{\sqrt{n}*l'(\theta)}{-l''(\theta)-\frac{l'''(\theta)}{2}(\phi-\theta)}\\=\frac{\frac{1}{\sqrt{n}}*l'(\theta)}{\frac{-l''(\theta)}{n}-\frac{l'''(\theta)}{2n}(\phi-\theta)} \tag{3}$$

(divide by n for numerator and denominator at the same time for the left hand side)

The let us see what is the numerator of the left side of (3):

$$\frac{1}{\sqrt{n}}l'(\theta)=\frac{1}{\sqrt{n}}\sum_{i=1}^n\frac{\partial logf(x_i;\theta)}{\partial \theta}$$

And note $$\frac{\partial logf(x_i;\theta)}{\partial \theta}$$ are i.i.d with variance $I(\theta)$ and $$E(\frac{\partial logf(x_i;\theta)}{\partial \theta})=0$$

$\therefore$ by CLT

$$\frac{1}{\sqrt{n}}l'(\theta)\sim \frac{1}{\sqrt{n}}N(0,nI(\theta))=N(0,I(\theta))$$

Next we will see what are in the denominator of (3):

$$-\frac{l''(\theta)}{n}=-\frac{1}{n}\sum_{i=1}^n\frac{\partial^2 logf(x_i;\theta)}{\partial \theta}\overset{P}{\rightarrow} I(\theta)$$ by Law of Large number.

For the term $$\frac{l'''(\theta)}{2n}(\phi-\theta)$$ in denominator of (3) we can prove it convergence in Probability to zero.

Finally, let us warp up everything

$$\sqrt{n}(\phi-\theta)\sim \frac{N(0,I(\theta)}{I(\theta)}=N(0,\frac{1}{I(\theta)})$$

This proved the theorem.

$\endgroup$
  • $\begingroup$ What makes you write $l'(\gamma)=l'(\theta)+\frac{l''(\theta)}{1!}(\gamma-\theta)^1+\frac{l'''(\theta)}{2!}(\gamma-\theta )^2$ and not $l'(\gamma)\approx l'(\theta)+\frac{l''(\theta)}{1!}(\gamma-\theta)^1+\frac{l'''(\theta)}{2!}(\gamma -\theta)^2$ ? Because essentially you are leaving out all the terms of the taylor expansion after the 2nd order. $\endgroup$ – Joogs Sep 30 '15 at 9:03
  • $\begingroup$ You are right, there are remainders left. Usually when people use Taylor Series they do not use $\approx$ but $=$ $\endgroup$ – Deep North Sep 30 '15 at 9:16
  • $\begingroup$ Are the remainder terms not important for the proof? $\endgroup$ – Joogs Sep 30 '15 at 9:19
  • $\begingroup$ No, they are not important for the proof. Even this part $\frac{l'''(\theta)}{2n}(\phi-\theta)$ will converge in probability to zero. $\endgroup$ – Deep North Sep 30 '15 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.