Help with Taylor expansion of log likelihood function

Question

I've reed the following part of a sketch of the proof that the maximum likelihood estimator is asymptotically normal:

"Sketch of the second part of the proof. Recall that we may write the likelihood equation as $$\sum_{i=0}^n U_i(\hat\theta)=0$$ where $ U_i(\phi)$ denotes the derivative of $ \log\! f(Y_i;\phi) $ with respect to $\phi$. Let $ U_i'(\phi) $ denote the derivative of $ \log\!f(Y_i;\phi)$ with respect to $ \phi$ . Now a Taylor expansion around $ \phi=\theta$ yields:

$$\sum_{i=0}^n U_i(\phi )-\sum_{i=0}^n U_i(\theta)\approx \left( \sum_{i=0}^n U_i'(\theta) \right)(\phi-\theta)$$"

This Taylor expansion does not make any sense to me. I am familiar with a Taylor expansion of $f(x) $ at $a$ as: $$\sum_{i=0}^\infty\frac{f^{(n)}(a)}{n!}(x-a)^n$$ I can see that all terms except the second term from the Taylor expansion of the log likelihood are excluded but I do not recognise where this term comes from as the original term involved subtracting two summations.

Answer 1

You should convince yourself that $f(x)-f(y)\approx f'(x)(x-y)$ is just another way to express Taylor expansion (under appropriate regularity assumptions).

Then, using linearity of the derivation, you can generalize it to the sum.

Answer 2

I think your problem is just the notation problem

Let us start from the beginning;

Suppose$X_1, X_2,...,X_n$ are i.i.d random variables with probability density function $f(x;\gamma)$

Usually, people will use $\theta$ here, I will keep it for later use.

The likelihood function is $L(\gamma;x)=f(x_1;\gamma)f(x_2;\gamma)...f(x_n;\gamma)$

The log likelihood function $l(\gamma;x)=log[f(x_1;\gamma)f(x_2;\gamma)...f(x_n;\gamma)]=\sum_{i=1}^nlogf(x_i;\gamma)$

As usual, we will take derivative of the log likelihood and set it to zero i.e $l'(\gamma;x)=0 $

or $\sum_{i=1}^nlog'f(x_i;\gamma)=0$

Here, you can see $U_i(\gamma)=log'f(x_i;\gamma)$

$\therefore l'(\gamma)=\sum_{i=1}^nU_i(\gamma)\tag{1}$ we omit $x$ here since the log likelihood function is a function of $\gamma $

Next we expand the function $l'(\gamma)$ into a Taylor series of order two abut $\theta$ .

$l'(\gamma)=l'(\theta)+\frac{l''(\theta)}{1!}(\gamma-\theta)^1+\frac{l'''(\theta)}{2!}(\gamma-\theta)^2$. This is the Taylor expanding for $l'(\gamma)$.

Next we evaluate the equation at $\phi$

$l'(\phi)=l'(\theta)+\frac{l''(\theta)}{1!}(\phi-\theta)^1+\frac{l'''(\theta)}{2!}(\phi-\theta)^2 \tag{2}$

Here you should see that $l'(\theta)=\sum_{i=1}^nU_i(\theta)$

and $l'(\phi)=\sum_{i=1}^nU_i(\phi)$

and $l''(\phi)=\sum_{i=1}^nU_i'(\phi)$

Ref (1)

If we ignore the third derivative term in (2) then your question will be answered here

We get that $\sum_{i=1}^n U_i(\phi )-\sum_{i=1}^n U_i(\theta)\approx \left( \sum_{i=1}^n U_i'(\theta) \right)(\phi-\theta)$

By the way I think $i$ usually start from $1$ not $0$ ,anyway it is just an index.

Let us don't stop here, we can go further to prove the theorem

We know that $l'(\phi)=0$

$\therefore l'(\theta)+\frac{l''(\theta)}{1!}(\phi-\theta)^1+\frac{l'''(\theta)}{2!}(\phi-\theta)^2=0$

i.e.

$l'(\theta)+(\phi-\theta)*[l''(\theta)+\frac{l'''(\theta)}{2}(\phi-\theta)]=0$

Next we rearrange the above terms:

$$(\phi-\theta)=\frac{l'(\theta)}{-l''(\theta)-\frac{l'''(\theta)}{2}(\phi-\theta)}$$

We multiply $\sqrt{n}$ for both side:

$$\sqrt{n}(\phi-\theta)=\frac{\sqrt{n}*l'(\theta)}{-l''(\theta)-\frac{l'''(\theta)}{2}(\phi-\theta)}\\=\frac{\frac{1}{\sqrt{n}}*l'(\theta)}{\frac{-l''(\theta)}{n}-\frac{l'''(\theta)}{2n}(\phi-\theta)} \tag{3}$$

(divide by n for numerator and denominator at the same time for the left hand side)

The let us see what is the numerator of the left side of (3):

$$\frac{1}{\sqrt{n}}l'(\theta)=\frac{1}{\sqrt{n}}\sum_{i=1}^n\frac{\partial logf(x_i;\theta)}{\partial \theta}$$

And note $$\frac{\partial logf(x_i;\theta)}{\partial \theta}$$ are i.i.d with variance $I(\theta)$ and $$E(\frac{\partial logf(x_i;\theta)}{\partial \theta})=0$$

$\therefore$ by CLT

$$\frac{1}{\sqrt{n}}l'(\theta)\sim \frac{1}{\sqrt{n}}N(0,nI(\theta))=N(0,I(\theta))$$

Next we will see what are in the denominator of (3):

$$-\frac{l''(\theta)}{n}=-\frac{1}{n}\sum_{i=1}^n\frac{\partial^2 logf(x_i;\theta)}{\partial \theta}\overset{P}{\rightarrow} I(\theta)$$ by Law of Large number.

For the term $$\frac{l'''(\theta)}{2n}(\phi-\theta)$$ in denominator of (3) we can prove it convergence in Probability to zero.

Finally, let us warp up everything

$$\sqrt{n}(\phi-\theta)\sim \frac{N(0,I(\theta)}{I(\theta)}=N(0,\frac{1}{I(\theta)})$$

This proved the theorem.