# Taylor's expansion on log likelihood

As far as I know, Taylors expansion works for fixed functions. I was wondering why it is justified to use it on the log likelihood. Even if we consider it as a function of only $\theta$, doesn't it have components that change as n increases (like $\sum X_i$ for example) ? Is it really always ok to say something like \begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)+ o(|\widehat{\theta} - \theta|) \\ \end{align*} Please help me understand why and when we can do something like this. Thanks in advance!

• If only Bourbaki had written a book about statistics... :) – Stéphane Laurent Aug 6 '14 at 18:55

## 3 Answers

If one includes the notational dependency on $n$: \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_n(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} we see that the puzzling point is the $n$-dependency of the $o$.

A rigorous way to get an approximate with a $n$-independent $o$: \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} is Taylor-Lagrange's inequality: if you are able to majorate $\ell_n'' \leq M$ uniformly in $n$ (on an appropriate interval) then you get the uniform $o$ by Taylor-Lagrange's inequality.

Strictly speaking, the likelihood function has two components: the observations and the parameters. It is typically seen as a function of the parameters when the sample is fixed but you can also study its behaviour as a random variable when you fix the parameters and see it as a function of the random variables (which are not fixed).

It is justified to use Taylor expansion when the sample is fixed, this is, a realisation of the corresponding random variables. The asymptotic behaviour is studied on a sequence of likelihood functions $\ell_n(\theta)$, indexed by the sample size in the usual way done in analysis.

The use of the Taylor expansion is actually quite common, since it allows for constructing a normal approximation to the likelihood by using a second order expansion as follows:

\begin{align*} \ell\left(\theta\right) & \approx \ell\left(\widehat{\theta}\right) + \frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right) + \frac{\partial^2\ell\left(\theta\right)}{\partial\theta^2}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)^2 , \\ \end{align*}

The first term is fixed, the second term is zero given that it is evaluated at its maximum. Then,

\begin{align*} \ell\left(\theta\right) & \approx C + K\left(\theta-\widehat{\theta}\right)^2, \\ \end{align*}

and finally taking exponential on both sides:

\begin{align*} {\mathcal L}\left(\theta\right) & \approx C^{\prime}\exp K \left(\theta-\widehat{\theta}\right)^2 \\ \end{align*}

which resembles the kernel of a normal density. $K$ is a negative constant since it is the second derivative evaluated at the MLE. The only requirement is, as in any other function, differentiability.

• But I slightly disagree with your comments because I see notes where they use a Taylors expansion and notation of convergence in probability and distribution. Thus people are not thinking of the sample as a fixed realization of the random variables. It seems there's a justification for expanding the likelihood and deriving convergence results but I can't find an explanation of it. – Stacy Aug 3 '14 at 14:05

Strictly speaking that expression doesn't make sense a priori. But It can be made precise. The log-likelihood is a random function (or a sequence of random functions if you're in the asymptotic setting) on the parameter space. So sure, for a given realization of that random function, one can write (for sample size $n$)

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|) \\ \end{align*}

exactly as what you have. But that is useless unless you know the random variable $|\widehat{\theta}_n - \theta|$ is small, say in probability as $n \rightarrow \infty$. In other words, you need that the MLE estimator is weakly consistent.

In other words,

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_p(1). \\ \end{align*}

Strictly speaking $l$ should be $l_n$. In the asymptotic setting, the log-likelihood is a squence of random functions but omitting $n$ is common.

• The problem of the OP is that $\ell=\ell_n$ and $\hat\theta=\hat\theta_n$ depend on $n$. Then the term $o(|\widehat{\theta}_n - \theta|)$ is somewhat strange because actually the $o$ depends on $n$ too. You have not adressed this question. – Stéphane Laurent Aug 5 '14 at 8:57
• That is what I said. Weak consistency of MLE means that the sequence is random variables $\hat{\theta_n} - \theta$ goes to $0$ in probability. That is how that expression in the question should be fixed. – Michael Aug 6 '14 at 7:39
• For any $C^1$-function, in this case, any realization of the log-likelihood function of some sample size (i.e. a realization of one element from a sequence of random functions), that expression holds. But being little-$o$ of something is useless unless you know that something goes to zero. Here the right notion of going to zero is, for example, almost surely or in probability. – Michael Aug 6 '14 at 8:00
• Yes but the main problem of the OP consists in uniformly majoring in $n$ the second-order derivative. – Stéphane Laurent Aug 6 '14 at 8:20
• That's how you read it, a little strange actually. OP merely asked for a precise statement. Where was $l''$ mentioned in the original question? Sure, for all $n$, $l''_n < M$ on a compact set of parameter space, for all its realizations modulo regularity conditions...that's trivial, and pretty useless. You can bound anything, here just by calculus, if you introduce a trivial compactness assumption. – Michael Aug 6 '14 at 9:01