# Proof that the log-likelihood is asymptotically quadratic

I was reading this article, where the author says that Maximum Likelihood (ML) estimates are asymptotically normal if the log-likelihood is asymptotically quadratic.

I have heard or read other times about the likelihood being asymptotically quadratic (under conditions), but I have never read any proof of this. Is anybody aware of such a proof?

It would be great to see a proof that shows also the rate of convergence $(\sqrt{n}?)$ of the log-likelihood to a quadratic function.

• Have you checked the book by LeCam and Yang referenced in the paper? Also, I think you mean to say that the ML are aymptotically normal (which they are) and not asymptotically quadratic. – Placidia Jan 5 '14 at 14:57
• Isn't that exactly what Theorem 3.1 (and corollary 3.2) in the paper prove? Le Cam theory is beautiful, but known to be challenging. Be sure that you are doing yourself justice by acquiring all the prerequisites. – tchakravarty Jan 5 '14 at 15:22
• @Placidia Thanks for the pointer to the typo. Unfortunately I don't have that book in my university's library, but maybe I can find it somewhere else. – Matteo Fasiolo Jan 5 '14 at 15:24
• @fgnu Maybe I have misunderstood the theorem, but to me it says that if the likelihood is quadratic then the distribution of $Z$ is normal, while I would like to see (if is exists) a result where the likelihood becomes (locally?) quadratic as the sample size increases. That might be against the spirit of the Geyer's paper, though! – Matteo Fasiolo Jan 5 '14 at 15:33
• I misunderstood your question. He does discuss convergence of the likelihood (ratio) to a quadratic function in section 4.3 using the appropriate definition of convergence of random functions given in 4.1 and 4.2. – tchakravarty Jan 5 '14 at 15:46

DISCUSSION: Local asymptotic quadraticity of the log-likelihood is proven under the same set of assumptions that prove the asymptotic normality of the ML estimator -it is not a prerequisite. The proof of MLE's asymptotic normality, given this set of assumptions, can be much more straightforward and short-and I provide it at the end. I combine two very different book sources, David Pollard's unpublished "Asymptopia" (ch. 2), and Hayashi's "Econometrics") (ch.7). I follow the notation of Hayashi. Moreover this is a "$n\rightarrow \infty$" kind of proof.

Assume a sample of size $n$ of i.i.d. observations coming from a distribution with density function $f_X(x;\theta_0)$ and use the following notation:
$$\ln f_X(x_i;\theta) = m(x_i;\theta),\;\; \frac {\partial m(x_i;\theta)}{\partial \theta}= s(x_i;\theta),\;\;\frac {\partial^2 m(x_i;\theta)}{\partial \theta\partial \theta}= H(x_i;\theta)$$ We have in turn, the log-likelihood, the score , and the Hessian, all related to observation $i$, not to the whole sample. Notation should be understood as reflecting vector-matrices (for more than one unknown parameter). Consider the average log-likelihood function (makes no difference in ML estimation, and strictly speaking, we should consider this likelihood in order for ML to be included in the M-estimators family)

$$\ell (\mathbf x;\theta) = \frac 1n \sum_{i=1}^nm(x_i;\theta)$$

Make the following assumptions that hold for a neighborhood of $\theta_0$, $\mathcal N(\theta_0)$:
[a] $\qquad m(x_i;\theta)$ is indeed twice differentiable, with second derivative continuous at $\theta_0$
[b] $\qquad$Its first derivative evaluated at $\theta_0$, $s(x_i;\theta_0)$ is square-integrable with respect to the probability measure involved (in other words, $E[s(x_i;\theta_0)s(x_i;\theta_0)']$ exists and is finite).
[c] $\qquad$ Its second derivative $H(x_i;\theta)$ is dominated by an integrable function in all $\mathcal N(\theta_0)$, (which means $E\left[\text {sup}_{\theta \in N(\theta_0)} ||H(x_i;\theta)||\right] <\infty$
[d] $\qquad$ $E[H(x_i;\theta)]$ is not a singular matrix (so its inverse exists)
[e] $\qquad$ $\theta_0$ is an interior point of the parameter space $\Theta$
[f] $\qquad$ There exists $\hat \theta: \hat \theta\xrightarrow{p}\theta_0$.

As Pollard remarks, the reason why we consider asymptotic normality of an estimator given that it is consistent, is that its asymptotic distribution would be of no use to us if the estimator was inconsistent.

LOCAL QUADRATICITY to go to ASYMPTOTIC NORMALITY.
We note that local quadraticity is proven for the sample average log-joint density, not for the sample log-likelihood -i.e. we are one step before viewing the sample log joint density as a likelihood function of $\theta$. But we will continue to call it "log-likelihood". In the above framework, the proof for local quadraticity essentially consists of nothing more than proving that the remainder from a 2nd order Taylor expansion of $m(x_i;\theta)$ around $\theta_0$ goes to zero asymptotically.
Assumption $[a]$ permits us to take this 2nd-order Taylor expansion and assumptions $[b]$ and $[c]$ permit to consider its expected value:

$$E[m(x_i;\theta)] = E[m(x_i;\theta_0)]+(\theta-\theta_0)E[s(x_i;\theta_0)]+\frac12(\theta-\theta_0)^2E[H(x_i;\theta_0)] + (\theta-\theta_0)^2E[R_2(x_i;\theta-\theta_0)]$$ where we have used the Peano form of the remainder term. Note that expectations relate to the $X$'s, the ML estimator has not yet been introduced.

Now, from a prior step, when proving the existence of a consistent estimator, by the "Identification Condition" we know that $E[m(x_i;\theta)]$ is at a maximum when $\theta =\theta_0$. Then due to assumption $[e]$ ($\theta_0$ is an interior point), this implies that $E[s(x_i;\theta_0)]=0$. If $\theta_0$ was not interior, then the Taylor expansion would retain a possibly non-zero linear term in $\theta$, and we would not be able to guarantee quadraticity (and this is why when $\theta_0$ is at the boundary the asymtptotic properties change).
Then one goes to show that the remainder converges uniformly in probability to zero (too much to write it down here), and we are left with

$$\ell (\mathbf x;\theta) = \frac 1n \sum_{i=1}^nm(x_i;\theta) \xrightarrow{p}E[m(x_i;\theta)] \xrightarrow{p} E[m(x_i;\theta_0)]+\frac12(\theta-\theta_0)^2E[H(x_i;\theta_0)] +o_p[(\theta-\theta_0)^2]$$ which is what is called "being locally asymptotically quadratic".

2nd step: Asymptotic Normality of the ML estimator
Keeping in mind the above results, we now take the 2nd-order Taylor expansion of the sample log-likelihood around $\theta_0$, manipulating the linear term,

$$\ell (x_i;\theta) = \ell (x_i;\theta_0) + \frac{(\theta-\theta_0)}{\sqrt n}\cdot\frac 1{\sqrt n}\sum_{i=1}^{n}s(x_i;\theta_0)+\frac12(\theta-\theta_0)^2\frac 1n\sum_{i=1}^{n}H(x_i;\theta_0) + R_n(x_i;(\theta-\theta_0))$$

Here too the remainder is shown to converge uniformly in probability to zero. If then we consider the derivative of the above with respect to $\theta$, and we set it equal to zero, we will obtain

$$\frac 1{\sqrt n}\cdot\frac 1{\sqrt n}\sum_{i=1}^{n}s(x_i;\theta_0)+(\hat \theta-\theta_0)\frac 1n\sum_{i=1}^{n}H(x_i;\theta_0)=0$$

where $\hat \theta$ is a "consistent root", which exists by assumption $[f]$.

Assumption $[d]$ permits to manipulate around and write

$${\sqrt n}(\hat \theta-\theta_0) = \left(-\frac 1n\sum_{i=1}^{n}H(x_i;\theta_0)\right)^{-1}\cdot\frac 1{\sqrt n}\sum_{i=1}^{n}s(x_i;\theta_0)$$

Assumption $[e]$ invoked previously gives us $E[s(x_i;\theta_0)]=0$. Assumption $[b]$ gives us that $E[s(x_i;\theta_0)s(x_i;\theta_0)']$ exists and is finite. Then the Central Limit Theorem holds and we have

$$\frac 1{\sqrt n}\sum_{i=1}^{n}s(x_i;\theta_0) \rightarrow_d N\left(0, E[s(x_i;\theta_0)s(x_i;\theta_0)']\right)$$

Finally, assumption $[c]$ guarantees that the inverse term is finite so

$${\sqrt n}(\hat \theta-\theta_0)\rightarrow_d N\left(0, (E[H(x_i;\theta_0)])^{-1}E[s(x_i;\theta_0)s(x_i;\theta_0)'](E[H(x_i;\theta_0)]^{-1})'\right)$$

DIRECT PROOF OF ASYMPTOTIC NORMALITY
Under the same set of assumptions, using the Mean-Value Theorem, we take a mean-value expansion of $\frac {\partial \ell(x_i;\hat \theta)}{\partial \theta}$ at $\theta_0$ and we obtain

$$\frac {\partial \ell(x_i;\hat \theta)}{\partial \theta}=\frac 1n\sum_{i=1}^{n}s(x_i;\theta_0)+(\hat \theta-\theta_0)\frac 1n\sum_{i=1}^{n}H(x_i;\bar \theta)=0$$ where $\bar \theta$ is a mean value between $\hat \theta$ and $\theta_0$. The assumptions permit us to write

$${\sqrt n}(\hat \theta-\theta_0) = \left(-\frac 1n\sum_{i=1}^{n}H(x_i;\bar \theta)\right)^{-1}\cdot\frac 1{\sqrt n}\sum_{i=1}^{n}s(x_i;\theta_0)$$

$\bar \theta$ is sandwiched between $\hat \theta$ and $\theta_0$ and it is also consistent for $\theta$. This and the assumptions, lead us to the same final result as before.

• Many thanks for your answer. One question: where can I find some info related to why the remainder in the first step converges to zero? Also, you write: "Now, we know that $m(x_i;\theta)$ is at a maximum when $\theta =\theta_0$", but I think that only the full likelihood $\ell (\mathbf x;\theta)$ is maximized (asymptotically) at $\theta_0$. – Matteo Fasiolo Apr 21 '14 at 9:01
• In the link I provide for Pollard's book, look for the .pdf "Classical" or "old-Classical" (but Pollard's style and notation are difficult). As for the second issue, what happens when we have a sample of size $n=1$? – Alecos Papadopoulos Apr 21 '14 at 9:31
• Thanks for the reference. Probably I'm missing something, but when $n = 1$ isn't $m(x_i, \theta)$ maximized at the MLE $\hat{\theta}$ rather than at the true parameter $\theta_0$? – Matteo Fasiolo Apr 21 '14 at 10:31
• Thanks, for insisting - I just re-stated the specific point, and now it is correctly stated. – Alecos Papadopoulos Apr 21 '14 at 11:13
• Have a look at the discussion here: andrewgelman.com/2012/07/05/… – kjetil b halvorsen Jun 10 '15 at 12:01