# Why is the asymptotic bias of the maximum likelihood estimate $b(\theta) = \frac{b_1(\theta)}{n}+\frac{b_2(\theta)}{n^2}+...$?

Firth (1993) states in his introduction that for a $$p$$-dimensional parameter $$\theta$$ the asymptotic bias of the maximum likelihood estimate $$\hat{\theta}$$ may be written as:

$$b(\theta) = \frac{b_1(\theta)}{n}+\frac{b_2(\theta)}{n^2} + ...$$,

where $$n$$ is the number of observations.

I understand why the first term is $$\frac{b_1(\theta)}{n}$$ from Cox and Snell (1968), but I can't find any references that explicitly derives the rest. Do the remaining terms of the bias shrink by order $$n$$ each time? I'm particularly struggling to understand exactly which terms of the Taylor expansion depend on $$n$$. If the estimated maximum likelihood $$\hat{\beta}$$ is the maximum of $$\begin{equation*} l(\beta) = \frac{1}{n}\sum_{i=1}^n \ln f(y_i|x_i; \beta), \end{equation*}$$ then why is the expected score, Hessian ext of order $$n$$? Some texts even have $$n^{1/2}$$ and I have no idea where they come from.

This is not exclusive to the maximum likelihood estimators.

In general, it is often a valid expansion of the expected value of an estimator, say $$\hat \theta,$$

$$\mathbb E\hat{\theta}=\theta +\frac{a_1}{n}+\frac{a_2}{n^2}+O(n^{-3}).$$

In fact, this form is used to reduce bias in stages aka Quenoullie's jackknife technique (cf. $$\rm [I]$$).

## Reference:

$$\rm [I]$$ Essential Statistical Inference: Theory and Methods, Dennis D. Books, L. A. Stefanski, Springer Science$$+$$Business, $$2013,$$ sec. $$10.1.2,$$ pp. $$386-388.$$

Thanks to the comment from User1865345, I found the answer in Quenoullie's jackknife paper. Here's a slightly modified explanation:

Given a series of observations $$y_1, y_2, ..., y_n$$, if the observations are taken in random order, the estimator $$\hat{\beta}$$ may be written as $$\begin{equation*} \hat{\beta} = \hat{\beta}(k_1, k_2, ..., k_m), \end{equation*}$$ where $$k_1, k_2, ..., k_m$$ are unbiased estimates of the cumulants $$\kappa_1, \kappa_2, ..., \kappa_m$$. Provided (i) $$m$$ is independent of $$n$$ (ii) The function $$\hat{\beta}$$ is capable of Taylorian expansion (iii) All of the cumulants are finite (iv) $$\hat{\beta}$$ is consistent, i.e. $$\beta = \lim_{n \rightarrow \infty} \hat{\beta}(k_1, ..., k_m)$$, it follows that $$\begin{equation*} \hat{\beta} - \beta = \sum_i (k_i - \kappa_i)\biggl(\frac{\partial \hat{\beta}}{\partial k_i}\biggl)_{k_i=\kappa_i} + \sum_i \sum_j (k_i - \kappa_i)(k_j - \kappa_j) \biggl(\frac{\partial^2 \hat{\beta}}{\partial k_i \partial k_j}\biggl)_{k_i=\kappa_i} + \hspace{0.1cm} ... \hspace{0.1cm}. \end{equation*}$$ Since the moments of the estimators, $$k_i$$, are power series in $$1/n$$, it follows that $$\mathbb{E}(\hat{\beta} - \beta)$$, i.e. the bias in $$\hat{\beta}$$, is also expressible as a power series in $$1/n$$. This means for a wide variety of statistics, it is true that the asymptotic bias can be written as $$\begin{equation*} b(\beta) = \frac{b_1(\beta)}{n} + \frac{b_2(\beta)}{n^2} + \frac{b_3(\beta)}{n^3} + ..., \end{equation*}$$ where $$n$$ is the sample size, assumed to be large relative to the number of parameters, $$k$$.

However, I don't understand how he interchanges $$k_i$$ cumulants for moments, any ideas?

• Cumulants and moments are homogeneous polynomial functions of each other.
– whuber
Commented Nov 19, 2023 at 16:20