Taylor's expansion on log likelihood As far as I know, Taylors expansion works for fixed functions. I was wondering why it is justified to use it on the log likelihood. Even if we consider it as a function of only $\theta$, doesn't it have components that change as n increases (like $\sum X_i$ for example) ? Is it really always ok to say something like
\begin{align*}
\ell\left(\theta\right) & = \ell\left(\widehat{\theta}\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)+ o(|\widehat{\theta} - \theta|)
\\
\end{align*}
Please help me understand why and when we can do something like this. Thanks in advance!
 A: If one includes the notational dependency on $n$:
$$
\begin{align*}
\ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_n(|\widehat{\theta}_n - \theta|^2)
\\
\end{align*}
$$
we see that the puzzling point is the $n$-dependency of the $o$.
A rigorous way to get an approximate with a $n$-independent $o$:
$$
\begin{align*}
\ell_n\left(\theta\right) & = \ell_n\left(\widehat{\theta}_n\right)+\frac{\partial\ell_n\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|^2)
\\
\end{align*}
$$
is Taylor-Lagrange's inequality: if you are able to majorate $\ell_n'' \leq M$ uniformly in $n$ (on an appropriate interval) then you get the uniform $o$ by Taylor-Lagrange's inequality.
A: Strictly speaking, the likelihood function has two components: the observations and the parameters. It is typically seen as a function of the parameters when the sample is fixed but you can also study its behaviour as a random variable when you fix the parameters and see it as a function of the random variables (which are not fixed).
It is justified to use Taylor expansion when the sample is fixed, this is, a realisation of the corresponding random variables. The asymptotic behaviour is studied on a sequence of likelihood functions $\ell_n(\theta)$, indexed by the sample size in the usual way done in analysis. 
The use of the Taylor expansion is actually quite common, since it allows for constructing a normal approximation to the likelihood by using a second order expansion as follows:
\begin{align*}
\ell\left(\theta\right) & \approx \ell\left(\widehat{\theta}\right) + \frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right) + \frac{\partial^2\ell\left(\theta\right)}{\partial\theta^2}\Bigr|_{\theta=\widehat{\theta}}\left(\theta-\widehat{\theta}\right)^2 ,
\\
\end{align*}
The first term is fixed, the second term is zero given that it is evaluated at its maximum. Then,
\begin{align*}
\ell\left(\theta\right) & \approx C + K\left(\theta-\widehat{\theta}\right)^2, 
\\
\end{align*}
and finally taking exponential on both sides:
\begin{align*}
{\mathcal L}\left(\theta\right) & \approx C^{\prime}\exp K \left(\theta-\widehat{\theta}\right)^2 
\\
\end{align*}
which resembles the kernel of a normal density. $K$ is a negative constant since it is the second derivative evaluated at the MLE. The only requirement is, as in any other function, differentiability.
A: Strictly speaking that expression doesn't make sense a priori. But It can be made precise. The log-likelihood is a random function (or a sequence of random functions if you're in the asymptotic setting) on the parameter space. So sure, for a given realization of that random function, one can write (for sample size $n$)
\begin{align*}
\ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o(|\widehat{\theta}_n - \theta|)
\\
\end{align*}
exactly as what you have. But that is useless unless you know the random variable $|\widehat{\theta}_n - \theta|$ is small, say in probability as $n \rightarrow \infty$. In other words, you need that the MLE estimator is weakly consistent. 
In other words,
\begin{align*}
\ell\left(\theta\right) & = \ell\left(\widehat{\theta}_n\right)+\frac{\partial\ell\left(\theta\right)}{\partial\theta}\Bigr|_{\theta=\widehat{\theta}_n}\left(\theta-\widehat{\theta}_n\right)+ o_p(1).
\\
\end{align*}
Strictly speaking $l$ should be $l_n$. In the asymptotic setting, the log-likelihood is a squence of random functions but omitting $n$ is common.
