1
$\begingroup$

Supposing I have a Poisson distributed random variable $X \sim \text{Poiss}(\lambda)$ with a parameter $\lambda$ that could take integer values only. Let $x$ be a single observation of a random variable. I could now get the estimate $\hat{\lambda}$ of the parameter by maximizing the likelihood function: $$ \hat{\lambda} = \arg\max_{\lambda \in \mathbb{N}}{ \log{p(x|\lambda)} }, \tag{1} $$ where $p(x|\lambda)$ is the Poisson PDF. To get the estimator variance $\text{Var}(\hat{\lambda})$ I first calculate the estimator distribution as $$ f(\hat{\lambda}|\lambda_0)=\sum_{x\in\mathbb{N}}{p(x|\lambda_0)\mathbf{1}_{\{\log{p(x|\hat{\lambda})} \geq \log{p(x|\lambda)}, \forall\lambda\in\mathbb{N}\}}}. \tag{2} $$ Here $\lambda_0\in\mathbb{N}$ is the true parameter value and $\mathbf{1}_{\{\cdot\}}$ is equal to 1 if condition in brackets holds and 0 otherwise. Since I computed (2) I can easily get $\text{Var}(\hat{\lambda})$. But things get worse when I try to estimate the parameter using a sample $x_1, x_2, \dots, x_n$ of the size $n$. In this case the estimator is $$ \hat{\lambda} = \arg\max_{\lambda \in \mathbb{N}}{ \sum_{j=1}^{n}{\log{p(x_j|\lambda)}} }. \tag{3} $$ Now the estimator PDF is $$ f(\hat{\lambda}|\lambda_0)=\sum_{x_1,\dots,x_n\in\mathbb{N}}{p(x_1|\lambda_0)\cdot\ldots\cdot p(x_n|\lambda_0)\mathbf{1}_{\{\sum_{j=1}^{n}{\log{p(x_j|\hat{\lambda})}} \geq \sum_{j=1}^{n}{\log{p(x_j|\lambda)}}, \forall\lambda\in\mathbb{N}\}}}. \tag{4} $$ There is no way to compute (4) for large $n$ (about $10^4$ in my case). So my questions are:

  1. Can I somehow simplify (4) to break the sum into a product?
  2. Are there some other ways to estimate the variance of the estimator (3)?

Things that I've tried

First of all I've tried to estimate the variance using Cramer-Rao bound by taking the finite difference derivative: $$ \text{Var}(\hat{\lambda}) = \left(n\mathbb{E}\left[(\log{p(x|\lambda_0+1)}-\log{p(x|\lambda_0))^2}\right]\right)^{-1}. \tag{5} $$ As expected, that didn't work quite well: the variance in simulation was lower than this value.

Then I came across a more general Hammersley–Chapman–Robbins bound. One can find that for my case the lower bound is (e.g. see [Dahiya R.C., Commun. Stat. - Theory Methods, 15(3), 709 (1986)]) $$ \text{Var}(\hat{\lambda}) = \left(e^{n/\lambda_0}-1\right)^{-1}. \tag{6} $$ However, this bound turned out to be too tight and was unreachable for any estimator I could find due to exponential decrease of the bound with $n$ (the same problem stated in Dahiya's article).

$\endgroup$

1 Answer 1

5
$\begingroup$

If you need the exact variance you're in trouble, I think. If not, progress can be made.

To start with, the monotone likelihood ratio property of exponential families means that $\hat\lambda$ is either the integer below the mean $\bar X$ or the integer above the mean. The variance of $\bar X$ is $\lambda/n$.

Consider some ranges of ($n$,$\lambda$):

  • If $\sqrt{\lambda/n}\gg 1$ then $\mathrm{var}[\hat\lambda]\approx \lambda/n$, since the reduction in variance caused by making $\hat\lambda$ an integer is small compared to the standard error of the mean.

  • For slightly smaller $\lambda/n$, $\mathrm{var}[\hat\lambda]$ is a bit smaller than $\lambda/n$. We can write $$\mathrm{var}[\hat\lambda] \approx \mathrm{var}[\bar X] - \mathrm{var}[\bar X-\hat\lambda]$$ and approximate the second term by the variance of a $U[-0.5, 0.5]$ $$\mathrm{var}[\hat\lambda] \approx \mathrm{var}[\bar X] -1/12$$ (this is the opposite of the Sheppard correction for rounding)

  • The difficult case: $\lambda/n$ not far from 1. I think this needs actual calculation, but only over the relatively small number of $\hat\lambda$ values with non-negligible probability.

  • At the far extreme, if $\sqrt{\lambda/n}\ll 1$, $\mathrm{var}[\hat\lambda]\approx 0$ (and even $n\mathrm{var}[\hat\lambda]\approx 0$), since $\hat\lambda=\lambda$ with very high probability. For example, if $\sqrt{\lambda/n}=1/10$, different integers are ten standard deviations apart, so it will be very unlikely for the closest integer to $\bar X$ not to be the true $\lambda$.

$\endgroup$
4
  • $\begingroup$ Can I interpret the last option as follows: for a continuous-valued parameter MLE asymptotically gives a normally distributed estimator, which I can then approximate to a discrete distribution by turning the PDF into a histogram with bin centers located at the discrete values? $\endgroup$
    – Krivoi
    Commented Jun 13, 2020 at 2:37
  • $\begingroup$ Yes, but the condition is stronger than just 'continuous-valued'. For some other type of distribution it might be possible for the discrete MLE to not just be a rounded continuous MLE, but for Poisson it will be. $\endgroup$ Commented Jun 13, 2020 at 2:43
  • $\begingroup$ Can I get some examples of a discrete parameter MLE? Is it like the one from this question? $\endgroup$
    – Krivoi
    Commented Jun 13, 2020 at 8:59
  • $\begingroup$ Yes, balls in urns are a classic example where the parameter is discrete. Another two are from ecology: estimating the population of a type of animal or the number of distinct species of animal. Often, though, you're in the first situation in the answer, where the discreteness can be ignored $\endgroup$ Commented Jun 14, 2020 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.