10
$\begingroup$

How can I construct an asymptotic confidence interval for a real parameter, starting from the MLE for that parameter?

$\endgroup$
  • $\begingroup$ One way to approach this problem is to use the delta method: en.wikipedia.org/wiki/Delta_method $\endgroup$ – user25658 Aug 29 '13 at 22:12
  • $\begingroup$ I noticed there are votes to close this question as too broad, but there is a general theorem about the asymptotic behaviour of MLEs that can be succinctly stated. I put in a terse answer that I'll expand a bit later. $\endgroup$ – Scortchi Aug 30 '13 at 7:14
13
$\begingroup$

Use the fact that for an i.i.d. sample of size $n$, given some regularity conditions, the MLE $\hat{\theta}$ is a consistent estimator of the true parameter $\theta_0$, & its distribution asymptotically Normal, with variance determined by the reciprocal of the Fisher information:

$$\sqrt{n}\left(\hat{\theta}-\theta_0\right) \rightarrow \mathcal{N}\left(0,\frac{1}{\mathcal{I}_1(\theta_0)}\right)$$ where $\mathcal{I}_1(\theta_0)$ is the Fisher information from a single sample. The observed information at the MLE $I(\hat{\theta})$ tends to the expected information asymptotically, so you can calculate (say 95%) confidence intervals with

$$\hat{\theta} \pm \frac{1.96}{\sqrt{nI_1\left(\hat\theta\right)}}$$

For example, if $X$ is a zero-truncated Poisson variate, you can get a formula for the observed information in terms of the MLE (which you have to calculate numerically): $$\newcommand{\e}{\mathrm{e}}\newcommand{\d}{\operatorname{d}}f(x) = \frac{\e^{-\theta}\theta^x}{x!(1-e^{-\theta})}$$

$$\ell(\theta)=-\theta+ x\log\theta -\log(1-\e^{-\theta})$$

$$\frac{\d\ell(\theta)}{\d\theta} = -1 +\frac{x}{\theta} - \frac{\e^{-\theta}}{1-\e^{-\theta}}$$

$$I_1\left(\hat{\theta}\right)=-\frac{\d^2\ell\left(\hat\theta\right)}{\left(\d\hat\theta\right)^2} = \frac{x}{\hat\theta} - \frac{\e^{-\hat\theta}}{\left(1-\e^{-\hat{\theta}}\right)^2}$$

Notable cases excluded by the regularity conditions include those where

  • the parameter $\theta$ determines the support of the data, e.g. sampling from a uniform distribution between nought and $\theta$
  • the number of nuisance parameters increases with sample size
$\endgroup$
  • $\begingroup$ Does this method apply unmodified when there are constraints on $\theta$, e.g. $\theta \in [0, 1]$? What about a MLE for $N$ parameters $\theta_i$, $i=0,...,N-1$ such that $\sum_{i=0}^{N-1} \theta_i = 1$ and $\theta_i \in [0,1]$? $\endgroup$ – quant_dev Dec 8 '14 at 10:47
  • 1
    $\begingroup$ If $\theta \in (0, 1)$, i.e. the true value isn't equal to one of the bounds. $\endgroup$ – Scortchi Dec 8 '14 at 11:33
  • $\begingroup$ If $\theta \in (0,1)$ and $\sigma(\hat{\theta}) > |\hat{\theta}|$, wouldn't it mean the normal approximation is not applicable and I need more samples? $\endgroup$ – quant_dev Dec 8 '14 at 12:16
  • $\begingroup$ Yes, it's only an asymptotic confidence interval. $\endgroup$ – Scortchi Dec 8 '14 at 12:21
  • $\begingroup$ For $\theta \in [0,1]$, a natural distribution would be Beta. Is there a similar theorem about how an MLE distribution can converge to Beta in this situation? $\endgroup$ – quant_dev Dec 8 '14 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.