# For what models does the bias of MLE fall faster than the variance?

Let $\hat\theta$ be a maximum likelihood estimate of a true parameter $\theta^*$ of some model. As the number of data points $n$ increases, the error $\lVert\hat\theta-\theta^*\rVert$ typically decreases as $O(1/\sqrt n)$. Using the triangle inequality and properties of the expectation, it's possible to show that this error rate implies that both the "bias" $\lVert \mathbb E\hat\theta - \theta^*\rVert$ and "deviation" $\lVert \mathbb E\hat\theta - \hat\theta\rVert$ decrease at the same $O(1/\sqrt{n})$ rate. Of course, it is possible for models to have bias that shrinks at a faster rate. Many models (like oridinary least squares regression) have no bias.

I'm interested in models that have bias that shrinks faster than $O(1/\sqrt n)$, but where the error does not shrink at this faster rate because the deviation still shrinks as $O(1/\sqrt n)$. In particular, I'd like to know sufficient conditions for a model's bias to shrink at the rate $O(1/n)$.

• Does $\lVert\hat\theta-\theta^*\rVert = (\hat\theta-\theta^*)^2$? Or? – Alecos Papadopoulos Jun 15 '17 at 13:50
• I was specifically asking about the L2 norm, yes. But I'd also be interested in other norms if it makes the question easier to answer. – Mike Izbicki Jun 15 '17 at 20:12
• $(\hat \theta -\theta^*)^2$ is $O_p(1/n)$. – Alecos Papadopoulos Jun 15 '17 at 20:14
• Sorry, I misread your comment. For the L2 norm in $d$ dimensions, $\Vert a-b\Vert = \sqrt{\sum_{i=1}^d (a_i-b_i)^2}$, and so convergence is at the rate of $O(1/\sqrt n)$. I agree that if we squared it then it would converge as $O(1/n)$. – Mike Izbicki Jun 15 '17 at 20:20
• Have you seen the ridge regression (Hoerl & Kennard 1970) paper ? I believe it gives conditions on the design matrix + penalty where this is expected to be true. – dcl Jun 15 '17 at 21:40

In general, you need models where the MLE is not asymptotically normal but converges to some other distribution (and it does so at a faster rate). This usually happens when the parameter under estimation is at the boundary of the parameter space. Intuitively, this means that the MLE will approach the parameter "only from the one side", so it "improves on convergence speed" since it is not "distracted" by going "back and forth" around the parameter.

A standard example, is the MLE for $\theta$ in an i.i.d. sample of $U(0,\theta)$ uniform r.v.'s The MLE here is the maximum order statistic,

$$\hat \theta_n = u_{(n)}$$

Its finite sample distribution is

$$F_{\hat \theta_n} = \frac {(\hat \theta_n)^n}{\theta ^n},\;\;\; f_{\hat \theta}=n\frac {(\hat \theta_n)^{n-1}}{\theta ^n}$$

$$\mathbb E(\hat \theta_n) = \frac {n}{n+1}\theta \implies B(\hat \theta) = -\frac {1}{n+1}\theta$$

So $B(\hat \theta_n) = O(1/n)$. But the same increased rate will hold also for the variance.

One can also verify that to obtain a limiting distribution, we need to look at the variable $n(\theta - \hat \theta_n)$,(i.e we need to scale by $n$) since

$$P[n(\theta - \hat \theta_n)\leq z] = 1-P[\hat \theta_n\leq \theta - (z/n)]$$

$$=1-\frac 1 {\theta^n}\cdot \left(\theta + \frac{-z}{n}\right)^n = 1-\frac {\theta^n} {\theta^n}\cdot \left(1 + \frac{-z/\theta}{n}\right)^n$$

$$\to 1- e^{-z/\theta}$$

which is the CDF of the Exponential distribution.

I hope this provides some direction.

• This is getting close, but I'm specifically interested in situations where the bias shrinks faster than the variance. – Mike Izbicki Jun 15 '17 at 21:24
• @MikeIzbicki Hmm... the bias convergence depends on the first moment of the distribution, and the (square root of the) variance is also a "first-order" magnitude. I am not sure then that this is possible to happen, because it appears that it would imply that the moments of the limiting distribution "arise" at convergence rates that are not compatible with each other... I' ll think about it though. – Alecos Papadopoulos Jun 15 '17 at 21:32

Following comments in my other answer (and looking again at the title of the OP's question!), here is an not very rigorous theoretical exploration of the issue.

We want to determine whether Bias $B(\hat \theta_n) = E(\hat \theta_n) - \theta$ may have different convergence rate than the square root of the Variance,

$$B(\hat \theta_n) = O(1/n^{\delta}),\;\;\; \sqrt {\text{Var}(\hat \theta_n)} = O(1/n^{\gamma}), \;\;\gamma \neq \delta \;???$$

We have

$$B(\hat \theta_n) = O(1/n^{\delta}) \implies \lim n^{\delta}\mathbb E(\hat \theta_n) < K \implies \lim n^{2\delta}[\mathbb E(\hat \theta_n)]^2 < K'$$

$$\implies [\mathbb E(\hat \theta_n)]^2 = O(1/n^{2\delta}) \tag{1}$$

while

$$\sqrt {\text{Var}(\hat \theta_n)} = O(1/n^{\gamma}) \implies \lim n^{\gamma}\sqrt{\mathbb E (\hat \theta_n^2) - [\mathbb E(\hat \theta_n)]^2 }<M$$

$$\implies \lim \sqrt{n^{2\gamma}\mathbb E (\hat \theta_n^2) - n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 }<M$$

$$\implies \lim n^{2\gamma}\mathbb E (\hat \theta_n^2) - \lim n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 < M' \tag{2}$$

We see that $(2)$ may hold happen if

A) both components are $O(1/n^{2\gamma})$, in which case we can only have $\gamma = \delta$.

B) But it may also hold if

$$\lim n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 \to 0 \implies [\mathbb E(\hat \theta_n)]^2 = o(1/n^{2\gamma}) \tag{3}$$

For $(3)$ to be compatible with $(1)$, we must have

$$n^{2\gamma} < n^{2\delta} \implies \delta > \gamma\tag {4}$$

So it appears that in principle it is possible to have the Bias converging at a faster rate than the square root of the variance. But we cannot have the square root of the variance converging at a faster rate than the Bias.

• How would you reconcile this with the existence of unbiased estimators like ordinary least squares? In that case, $B(\hat\theta)=0$, but $\sqrt{Var(\hat\theta)} = O(1/\sqrt n)$. – Mike Izbicki Jun 15 '17 at 23:45
• @MikeIzbicki Is the concept of convergence/big-O applicable in this case? Because here $B(\hat \theta)$ is not "$O()$-anything" to begin with. – Alecos Papadopoulos Jun 16 '17 at 0:12
• In this case, $\mathbb E\hat\theta=\theta^*$, so $B(\hat\theta) = \lVert \mathbb E \hat\theta - \theta^*\rVert = 0 = O(1) = O(1/n^0)$. – Mike Izbicki Jun 16 '17 at 5:04
• @MikeIzbicki But also $B(\hat \theta) = O(n)$ or $B(\hat \theta) =O(1/\sqrt{n})$ or any other you care to write down. So which one is the rate of convergence here? – Alecos Papadopoulos Jun 16 '17 at 8:16
• @MikeIzbicki I have corrected my answer to show that it is possible in principle to have the Bias converging faster, although I still think the "zero-bias" example is problematic. – Alecos Papadopoulos Jun 16 '17 at 9:11