6
$\begingroup$

As far as I know, Taylors expansion works for fixed functions. I was wondering why it is justified to use it on the log likelihood. Even if we consider it as a function of only $\theta$, doesn't it have components that change as n increases (like $\sum X_i$ for example) ? Is it really always ok to say something like \begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)+ o(|\widehat{\theta} - \theta|) \\ \end{align*} Please help me understand why and when we can do something like this. Thanks in advance!

$\endgroup$
  • 1
    $\begingroup$ If only Bourbaki had written a book about statistics... :) $\endgroup$ – Stéphane Laurent Aug 6 '14 at 18:55
4
$\begingroup$

If one includes the notational dependency on $n$: $$ \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_n(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} $$ we see that the puzzling point is the $n$-dependency of the $o$.

A rigorous way to get an approximate with a $n$-independent $o$: $$ \begin{align*} \ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|^2) \\ \end{align*} $$ is Taylor-Lagrange's inequality: if you are able to majorate $\ell_n'' \leq M$ uniformly in $n$ (on an appropriate interval) then you get the uniform $o$ by Taylor-Lagrange's inequality.

$\endgroup$
2
$\begingroup$

Strictly speaking, the likelihood function has two components: the observations and the parameters. It is typically seen as a function of the parameters when the sample is fixed but you can also study its behaviour as a random variable when you fix the parameters and see it as a function of the random variables (which are not fixed).

It is justified to use Taylor expansion when the sample is fixed, this is, a realisation of the corresponding random variables. The asymptotic behaviour is studied on a sequence of likelihood functions $\ell_n(\theta)$, indexed by the sample size in the usual way done in analysis.

The use of the Taylor expansion is actually quite common, since it allows for constructing a normal approximation to the likelihood by using a second order expansion as follows:

\begin{align*} \ell\left(\theta\right) & \approx \ell\left(\widehat{\theta}\right) + \frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right) + \frac{\partial^2\ell\left(\theta\right)}{\partial\theta^2}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)^2 , \\ \end{align*}

The first term is fixed, the second term is zero given that it is evaluated at its maximum. Then,

\begin{align*} \ell\left(\theta\right) & \approx C + K\left(\theta-\widehat{\theta}\right)^2, \\ \end{align*}

and finally taking exponential on both sides:

\begin{align*} {\mathcal L}\left(\theta\right) & \approx C^{\prime}\exp K \left(\theta-\widehat{\theta}\right)^2 \\ \end{align*}

which resembles the kernel of a normal density. $K$ is a negative constant since it is the second derivative evaluated at the MLE. The only requirement is, as in any other function, differentiability.

$\endgroup$
  • 3
    $\begingroup$ But I slightly disagree with your comments because I see notes where they use a Taylors expansion and notation of convergence in probability and distribution. Thus people are not thinking of the sample as a fixed realization of the random variables. It seems there's a justification for expanding the likelihood and deriving convergence results but I can't find an explanation of it. $\endgroup$ – Stacy Aug 3 '14 at 14:05
-1
$\begingroup$

Strictly speaking that expression doesn't make sense a priori. But It can be made precise. The log-likelihood is a random function (or a sequence of random functions if you're in the asymptotic setting) on the parameter space. So sure, for a given realization of that random function, one can write (for sample size $n$)

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|) \\ \end{align*}

exactly as what you have. But that is useless unless you know the random variable $|\widehat{\theta}_n - \theta|$ is small, say in probability as $n \rightarrow \infty$. In other words, you need that the MLE estimator is weakly consistent.

In other words,

\begin{align*} \ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_p(1). \\ \end{align*}

Strictly speaking $l$ should be $l_n$. In the asymptotic setting, the log-likelihood is a squence of random functions but omitting $n$ is common.

$\endgroup$
  • 2
    $\begingroup$ The problem of the OP is that $\ell=\ell_n$ and $\hat\theta=\hat\theta_n$ depend on $n$. Then the term $o(|\widehat{\theta}_n - \theta|)$ is somewhat strange because actually the $o$ depends on $n$ too. You have not adressed this question. $\endgroup$ – Stéphane Laurent Aug 5 '14 at 8:57
  • $\begingroup$ That is what I said. Weak consistency of MLE means that the sequence is random variables $\hat{\theta_n} - \theta$ goes to $0$ in probability. That is how that expression in the question should be fixed. $\endgroup$ – Michael Aug 6 '14 at 7:39
  • $\begingroup$ For any $C^1$-function, in this case, any realization of the log-likelihood function of some sample size (i.e. a realization of one element from a sequence of random functions), that expression holds. But being little-$o$ of something is useless unless you know that something goes to zero. Here the right notion of going to zero is, for example, almost surely or in probability. $\endgroup$ – Michael Aug 6 '14 at 8:00
  • $\begingroup$ Yes but the main problem of the OP consists in uniformly majoring in $n$ the second-order derivative. $\endgroup$ – Stéphane Laurent Aug 6 '14 at 8:20
  • $\begingroup$ That's how you read it, a little strange actually. OP merely asked for a precise statement. Where was $l''$ mentioned in the original question? Sure, for all $n$, $l''_n < M$ on a compact set of parameter space, for all its realizations modulo regularity conditions...that's trivial, and pretty useless. You can bound anything, here just by calculus, if you introduce a trivial compactness assumption. $\endgroup$ – Michael Aug 6 '14 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.