# Maximum Likelihood estimator - confidence interval

How can I construct an asymptotic confidence interval for a real parameter, starting from the MLE for that parameter?

• One way to approach this problem is to use the delta method: en.wikipedia.org/wiki/Delta_method – user25658 Aug 29 '13 at 22:12
• I noticed there are votes to close this question as too broad, but there is a general theorem about the asymptotic behaviour of MLEs that can be succinctly stated. I put in a terse answer that I'll expand a bit later. – Scortchi Aug 30 '13 at 7:14

Use the fact that for an i.i.d. sample of size $n$, given some regularity conditions, the MLE $\hat{\theta}$ is a consistent estimator of the true parameter $\theta_0$, & its distribution asymptotically Normal, with variance determined by the reciprocal of the Fisher information:

$$\sqrt{n}\left(\hat{\theta}-\theta_0\right) \rightarrow \mathcal{N}\left(0,\frac{1}{\mathcal{I}_1(\theta_0)}\right)$$ where $\mathcal{I}_1(\theta_0)$ is the Fisher information from a single sample. The observed information at the MLE $I(\hat{\theta})$ tends to the expected information asymptotically, so you can calculate (say 95%) confidence intervals with

$$\hat{\theta} \pm \frac{1.96}{\sqrt{nI_1\left(\hat\theta\right)}}$$

For example, if $X$ is a zero-truncated Poisson variate, you can get a formula for the observed information in terms of the MLE (which you have to calculate numerically): $$\newcommand{\e}{\mathrm{e}}\newcommand{\d}{\operatorname{d}}f(x) = \frac{\e^{-\theta}\theta^x}{x!(1-e^{-\theta})}$$

$$\ell(\theta)=-\theta+ x\log\theta -\log(1-\e^{-\theta})$$

$$\frac{\d\ell(\theta)}{\d\theta} = -1 +\frac{x}{\theta} - \frac{\e^{-\theta}}{1-\e^{-\theta}}$$

$$I_1\left(\hat{\theta}\right)=-\frac{\d^2\ell\left(\hat\theta\right)}{\left(\d\hat\theta\right)^2} = \frac{x}{\hat\theta} - \frac{\e^{-\hat\theta}}{\left(1-\e^{-\hat{\theta}}\right)^2}$$

Notable cases excluded by the regularity conditions include those where

• the parameter $\theta$ determines the support of the data, e.g. sampling from a uniform distribution between nought and $\theta$
• the number of nuisance parameters increases with sample size
• Does this method apply unmodified when there are constraints on $\theta$, e.g. $\theta \in [0, 1]$? What about a MLE for $N$ parameters $\theta_i$, $i=0,...,N-1$ such that $\sum_{i=0}^{N-1} \theta_i = 1$ and $\theta_i \in [0,1]$? – quant_dev Dec 8 '14 at 10:47
• If $\theta \in (0, 1)$, i.e. the true value isn't equal to one of the bounds. – Scortchi Dec 8 '14 at 11:33
• If $\theta \in (0,1)$ and $\sigma(\hat{\theta}) > |\hat{\theta}|$, wouldn't it mean the normal approximation is not applicable and I need more samples? – quant_dev Dec 8 '14 at 12:16
• Yes, it's only an asymptotic confidence interval. – Scortchi Dec 8 '14 at 12:21
• For $\theta \in [0,1]$, a natural distribution would be Beta. Is there a similar theorem about how an MLE distribution can converge to Beta in this situation? – quant_dev Dec 8 '14 at 12:28