2
$\begingroup$

Preliminaries

My question came about when reading Lawrence and Chromy's paper on Maximum Likelihood (ML) estimation of histograms (link). I am aware of this question, but I fail to see how that helps me. Please provide additional info if you were to link me to this answer. In their supplementary information L&C state

$n$ is large enough that the probability that an event is with in any bin is small, leading to a Poisson distribution rather than a multinomial distribution.

By $n$ they mean the number of histogram bins. Why would they even mention that? Here is my trouble with that statement:

Histogram as Multinomial

They observe independent identically distributed random events and bin them in a histogram with $n$ bins with index $j=1,...,n$. Let

$prob(b_j|\theta)$: Probability of observing an event in bin with index $j$.

This is given by a theoretical model that depends on parameter(s) $\theta$ to be estimated by ML. Furthermore let

$$\sum_{j=1}^n prob(b_j|\theta) = 1$$

independent of $\theta$, which is relevant to my question. Then (I think) the likelihood of observing a histogram with $m_j$ counts in bin $j=1,...,n$ is given by a multinomial distribution

$$L_M(\theta) := prob(\{m_j\}|\theta) = \frac{M!}{\prod_{j=1}^n m_j!}\prod_{j=1}^n prob(b_j|\theta)^{m_j}$$

which gives an expected value $\mu_j(\theta)=M\cdot prob(b_j|\theta)$ for each bin, where $M=\sum_{j=1}^n m_j$ is the total number of counts in the histogram.

Poisson Distribution

Lawrence and Cromy model the number of counts in each bin as a Poisson Distribution. The likelihood of observing $m_j$ counts in one specific bin $j$ is

$$prob(m_j|\theta) = \frac{\mu_j(\theta)^{m_j}}{m_j!}\exp(-\mu_j)$$

Now by the independence they can write the likelihood of the set $\{n_j\}$ as

$$L_P(\theta) := prob(\{m_j\}|\theta) = \prod_{j=1}^n\frac{\mu_j(\theta)^{m_j}}{m_j!}\exp(-\mu_j)$$.

My Trouble

This is where I get confused. Why don't L&C just do this calculation:

$$\begin{eqnarray}L_P(\theta)&=& \prod_{j=1}^n\frac{\mu_j(\theta)^{m_j}}{m_j!}\prod_{j=1}^n\exp(-\mu_j(\theta)) \\ &=&\prod_{j=1}^n\frac{\mu_j(\theta)^{m_j}}{m_j!}\exp\left(-\sum_{j=1}^n\mu_j(\theta)\right) \\ &=& \exp(-M)\cdot \prod_{j=1}^n\frac{\mu_j(\theta)^{m_j}}{m_j!} \end{eqnarray}$$ and ultimately after disregarding terms that are not dependent on $\theta$

$$L_P(\theta)\propto \prod_{j=1}^n prob(b_j|\theta)^{m_j}$$

which is analogous to the multinomial likelihood where also $$L_M(\theta)\propto \prod_{j=1}^n prob(b_j|\theta)^{m_j}$$

So maximizing either likelihood should give the same ML estimate for $\theta$...

My questions are

  • Is my reasoning flawed?
  • Where is the difference between using the Multinomial and the Poisson model here?
  • Why would L&C require the probability of an event occuring in any one bin being small?

All help is greatly appreciated.

$\endgroup$
1
  • $\begingroup$ I have a slightly sideways point to make. If you consider an experiment which produces numerical values, and you histogram those values, and fix the number of bins in the histogram, and fix the number of numerical values generated, then why is the resulting histogram containing posisson distributed data? Since the total number of events in the histogram is fixed, surely the bins are no longer independent? $\endgroup$ Commented Nov 5, 2023 at 13:54

2 Answers 2

1
$\begingroup$

1. $$L_M(\theta) := prob(\{m_j\}|\theta) = \frac{M!}{\prod_{j=1}^n m_j!}\prod_{j=1}^{n-1} prob(b_j|\theta)^{m_j}\left(1-\sum_{i=1}^{n-1}prob(b_j|\theta)\right)^{m_n}$$

  1. Two distributions are different. One specific difference is we assume that $m_j$ are independent if $m_j$ follows Poisson, and it is reasonable in most real world. But if $m_i$ follows multinomial, $m_j$ and $m_i$ negative correlation.

  2. By increasing n, the probability of an event occuring in any one bin being small, such that multinnomial distribution is very similar to Poisson distribution. For examaple, the negative correlation between $m_i$ follows multinomial, $m_j$ and $m_i$ decreases to ignorable degree.

$\endgroup$
4
  • $\begingroup$ Thanks, I think I almost get it. I feel that the correlation of the $\{m_j\}$ is the key fact I missed. Could you clarify some points: I have never seen the functional form of the multinomial distribution like that. Where does that come from? Is that to incorporate the correlation of the counts? $\endgroup$
    – geo
    Commented Oct 24, 2018 at 6:36
  • $\begingroup$ Especially: what does $m_n$ refer to in the sum and where does the $(1-\sum ...)$ term come from? $\endgroup$
    – geo
    Commented Oct 24, 2018 at 7:18
  • $\begingroup$ In my equation, there are m-1 parameters, and in yours, there are m parameters. Of course, $m_n$ can be $\sum_{i=1}^n - \sum_{i=1}^{n-1}$. For multinomial, your formula is OK, but need to add two conditions, that sum of probabilities = 1 and sum of $m_i$ = m$(total # of trials). Otherwise, you will go wrong in math proof. $\endgroup$
    – user158565
    Commented Oct 24, 2018 at 12:49
  • $\begingroup$ Thanks, I get it now. I got confused with my own notation. $\endgroup$
    – geo
    Commented Oct 24, 2018 at 18:18
0
$\begingroup$

In many cases, the number of entries in a bin follows a Poisson distribution. I will explain with an example.

Suppose we are observing a star with a telescope and want to make a histogram with the energy of the incoming photons. Consider a single energy bin limited by energies $E_i$ and $E_{i+1}$. The probability that a photon emitted by the star arrives at the telescope during the observation time and has energy between $E_i$ and $E_{i+1}$ is very low. This situation fits the hypothesis of the "law of rare events". Then, the number of entries in the bin ($k_i$) follows a Poisson distribution.

We can repeat the reasoning for the number of entries in another bin j ($k_j$). The variables $k_i$ and $k_j$ are independent as the probability of one photon reaching the telescope does not depend on the probability of another photon.

The total number of photons in the histogram ($N$) is the sum of the number of photons in all bins, $N=\sum_i k_i$. As $k_i$ is Poissonian, therefore so does the sum of them, the random variable $N$. This makes sense as we could have applied the original reasoning to the same histogram with only one bin.

So, there are two types of histograms. Histograms modeled with a multinomial distribution have a fixed $N$. This will correspond to an experiment that stops after $N$ entries are measured. In turn, histograms modeled with Poisson have a variable $N$. This will be useful for example when we measure for a fixed time. The second case is very common in many scientific experiments. The book Statistical Data Analysis by Glen Cowan has a very nice description of both histogram types.

The Poisson reasoning is also valid if the probability of a single bin is large relative to the probability of the full histogram. This is picked up in data when a single bin has a significant portion of the histogram entries. I mention this because the approximation of a multinomial by Poisson is only possible when the probability of a single bin is small.

Another consideration is that a chi-square test for a fixed time histogram has to use k degrees of freedom with k the number of bins, instead of the $k-1$ used for fixed entries histograms.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.