Why is the asymptotic bias of the maximum likelihood estimate $b(\theta) = \frac{b_1(\theta)}{n}+\frac{b_2(\theta)}{n^2}+...$?

Question

Firth (1993) states in his introduction that for a $p$-dimensional parameter $\theta$ the asymptotic bias of the maximum likelihood estimate $\hat{\theta}$ may be written as:

$b(\theta) = \frac{b_1(\theta)}{n}+\frac{b_2(\theta)}{n^2} + ...$,

where $n$ is the number of observations.

I understand why the first term is $\frac{b_1(\theta)}{n}$ from Cox and Snell (1968), but I can't find any references that explicitly derives the rest. Do the remaining terms of the bias shrink by order $n$ each time? I'm particularly struggling to understand exactly which terms of the Taylor expansion depend on $n$. If the estimated maximum likelihood $\hat{\beta}$ is the maximum of \begin{equation*} l(\beta) = \frac{1}{n}\sum_{i=1}^n \ln f(y_i|x_i; \beta), \end{equation*} then why is the expected score, Hessian ext of order $n$? Some texts even have $n^{1/2}$ and I have no idea where they come from.

Answer 1

This is not exclusive to the maximum likelihood estimators.

In general, it is often a valid expansion of the expected value of an estimator, say $\hat \theta, $

$$\mathbb E\hat{\theta}=\theta +\frac{a_1}{n}+\frac{a_2}{n^2}+O(n^{-3}).$$

In fact, this form is used to reduce bias in stages aka Quenoullie's jackknife technique (cf. $\rm [I]$).

---

Reference:

$\rm [I]$ Essential Statistical Inference: Theory and Methods, Dennis D. Books, L. A. Stefanski, Springer Science$+$Business, $2013, $ sec. $10.1.2, $ pp. $386-388.$

Answer 2

Thanks to the comment from User1865345, I found the answer in Quenoullie's jackknife paper. Here's a slightly modified explanation:

Given a series of observations $y_1, y_2, ..., y_n$, if the observations are taken in random order, the estimator $\hat{\beta}$ may be written as \begin{equation*} \hat{\beta} = \hat{\beta}(k_1, k_2, ..., k_m), \end{equation*} where $k_1, k_2, ..., k_m$ are unbiased estimates of the cumulants $\kappa_1, \kappa_2, ..., \kappa_m$. Provided (i) $m$ is independent of $n$ (ii) The function $\hat{\beta}$ is capable of Taylorian expansion (iii) All of the cumulants are finite (iv) $\hat{\beta}$ is consistent, i.e. $\beta = \lim_{n \rightarrow \infty} \hat{\beta}(k_1, ..., k_m)$, it follows that \begin{equation*} \hat{\beta} - \beta = \sum_i (k_i - \kappa_i)\biggl(\frac{\partial \hat{\beta}}{\partial k_i}\biggl)_{k_i=\kappa_i} + \sum_i \sum_j (k_i - \kappa_i)(k_j - \kappa_j) \biggl(\frac{\partial^2 \hat{\beta}}{\partial k_i \partial k_j}\biggl)_{k_i=\kappa_i} + \hspace{0.1cm} ... \hspace{0.1cm}. \end{equation*} Since the moments of the estimators, $k_i$, are power series in $1/n$, it follows that $\mathbb{E}(\hat{\beta} - \beta)$, i.e. the bias in $\hat{\beta}$, is also expressible as a power series in $1/n$. This means for a wide variety of statistics, it is true that the asymptotic bias can be written as \begin{equation*} b(\beta) = \frac{b_1(\beta)}{n} + \frac{b_2(\beta)}{n^2} + \frac{b_3(\beta)}{n^3} + ..., \end{equation*} where $n$ is the sample size, assumed to be large relative to the number of parameters, $k$.

However, I don't understand how he interchanges $k_i$ cumulants for moments, any ideas?