Taylor's expansion on log likelihood

Question

As far as I know, Taylors expansion works for fixed functions. I was wondering why it is justified to use it on the log likelihood. Even if we consider it as a function of only $\theta$, doesn't it have components that change as n increases (like $\sum X_i$ for example) ? Is it really always ok to say something like \begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)+ o(|\widehat{\theta} - \theta|) \\ \end{align*} Please help me understand why and when we can do something like this. Thanks in advance!

Answer 1

If one includes the notational dependency on $n$: $$ \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_n(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} $$ we see that the puzzling point is the $n$-dependency of the $o$.

A rigorous way to get an approximate with a $n$-independent $o$: $$ \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} $$ is Taylor-Lagrange's inequality: if you are able to majorate $\ell_n'' \leq M$ uniformly in $n$ (on an appropriate interval) then you get the uniform $o$ by Taylor-Lagrange's inequality.

Answer 2

Strictly speaking, the likelihood function has two components: the observations and the parameters. It is typically seen as a function of the parameters when the sample is fixed but you can also study its behaviour as a random variable when you fix the parameters and see it as a function of the random variables (which are not fixed).

It is justified to use Taylor expansion when the sample is fixed, this is, a realisation of the corresponding random variables. The asymptotic behaviour is studied on a sequence of likelihood functions $\ell_n(\theta)$, indexed by the sample size in the usual way done in analysis.

The use of the Taylor expansion is actually quite common, since it allows for constructing a normal approximation to the likelihood by using a second order expansion as follows:

\begin{align*} \ell\left(\theta\right) & \approx \ell\left(\widehat{\theta}\right) + \frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right) + \frac{\partial^2\ell\left(\theta\right)}{\partial\theta^2}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)^2 , \\ \end{align*}

The first term is fixed, the second term is zero given that it is evaluated at its maximum. Then,

\begin{align*} \ell\left(\theta\right) & \approx C + K\left(\theta-\widehat{\theta}\right)^2, \\ \end{align*}

and finally taking exponential on both sides:

\begin{align*} {\mathcal L}\left(\theta\right) & \approx C^{\prime}\exp K \left(\theta-\widehat{\theta}\right)^2 \\ \end{align*}

which resembles the kernel of a normal density. $K$ is a negative constant since it is the second derivative evaluated at the MLE. The only requirement is, as in any other function, differentiability.

Answer 3

Strictly speaking that expression doesn't make sense a priori. But It can be made precise. The log-likelihood is a random function (or a sequence of random functions if you're in the asymptotic setting) on the parameter space. So sure, for a given realization of that random function, one can write (for sample size $n$)

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|) \\ \end{align*}

exactly as what you have. But that is useless unless you know the random variable $|\widehat{\theta}_n - \theta|$ is small, say in probability as $n \rightarrow \infty$. In other words, you need that the MLE estimator is weakly consistent.

In other words,

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_p(1). \\ \end{align*}

Strictly speaking $l$ should be $l_n$. In the asymptotic setting, the log-likelihood is a squence of random functions but omitting $n$ is common.