3
$\begingroup$

The following lines are taken from a lecture found here. I have a very small question regarding this explanation of Poisson likelihood.

The Poisson likelihood can be used any time your data come in discrete intervals (which we'll call “counts”), and the counts are independent of each other. Schematically, we imagine dividing data space up into “bins”, which could be bins in energy channel of our detector, location on the sky, time of arrival, or any of a number of other things. Suppose that in a particular model $m$, you expect there to be $m_i$ counts in bin $i$. Then if the model is correct the likelihood of actually observing $d_i$ counts in bin $i$ of the data is, from the Poisson distribution,

$L_i = m_i^{d_i}/d_i! \; exp(-m_i)$

Note that $m_i$ can be any positive real number, whereas $d_i$ must be an integer. Note also that the sum of $L_i$ from $d_i = 0$ to ∞ is 1. The likelihood for the whole data set is the product of the likelihoods for each bin:

$L = \Pi\; m_i^{d_i}/d_i! \; exp(-m_i)$

This becomes better and better approximated by a Gaussian as $m_i$ increases. (...) If one uses Poisson likelihoods, small numbers are fine. In fact, if you can manage, the best way to represent your data is to have bins so tiny that you expect either 0 or 1 count per bin.

The question I have is: what happens if one of the expected values $m_i$ is zero (ie: the model predicts zero counts in that bin $i$)? Wouldn't that make the final $L$ value null?

Edit

After reading @leonbloy and @Henry's answer I think I should explain a bit more my particular problem with this analysis and what it is that I'm clearly not understanding.

In my case I have two histograms representing the modeled data and the observed data. So the $[d_i, \; i=1,n]$ histogram is made out of actual observations (ie: the counts in each bin $d_i$ are obtained through an experiment) and the histogram $[m_i, \; i=1,n]$ is produced by a model: this would mean that the $m_i$ counts in my case are integers that can actually take the value 0.

Now, this is where I'm clearly getting something wrong because I read in that article a line that says "Note that $m_i$ can be any positive real number" and @leonbloy's answer also points in that direction, when in my case the $m_i$ values are all integers and do take the value 0 a lot of times.

I must be skipping a crucial step concerning the calculation of the $m_i$ values. So how should I calculate the $m_i$ values if my model gives me counts per bin?

@Henry's answer seems to contradict @leonbloy's answer since he says $m_i=0$ is a possibility whereas @leonbloy says it is not.

In any case, Henry seems to be assuming the whenever $m_i=0$ then $d_i=0$ also so that one obtains $L_i=1$ because $0⁰=1$. This is not my case, since $m_i$ could be 0 but it doesn't mean $d_i$ is forced to be 0 also (my observed histogram, $[d_i , \; i=1,n]$, and my modeled histogram, $[m_i , \; i=1,n]$, are derived independently of each other)

Edit 2

I think I'm beginning to understand, but I'm still a bit lost. Please take a look at this excerpt taken from this paper:

(...) Therefore the selected test to perform on these histogram is a log likelihood test for Poisson statistics (Eidelman et al. 2004):

$-2ln(\lambda(\theta))=2\, \sum\limits_{i=1}^N \, (\nu_i(\theta)-n_i+n_i\, ln\frac{n_i}{\nu_i(\theta)})$ (8)

In this formula $\theta$ is the set of unknown parameters one wants to derive, $n = (n_1 , n_2, . . . , n_N)$ is the data vector containing the observations with $N$ the number of bins in a histogram. $\nu$ are the expected values, which are derived from the histograms of the modelled data and are therefore dependent on $\theta$. When $n_i = 0$, the last term in Eq. (8) is set to zero (...)

As you can see, the analysis is the same (the difference here is that the last equation shows the likelihood ratio), now the expected values are $\nu_i(\theta)$.

From what I've understood so far that parameter $\nu_i(\theta)$ should be independent of the number of counts in the bins and have just one value for each histogram. If this is so then why use the $i$ subscript? And how should I calculate that value for each histogram?

$\endgroup$
  • $\begingroup$ I think you are confusing $d_i$ (integer-valued random variable) with $m_i$ (expectation of $d_i$, real, not a random variable) . You cannot estimate $m_i$ with one measure. Either you assume all your bins have same expectation (say, your "space" is homogeneous, or your process is stationary) and then you estimate $m_i$ (same for all $i$) averaging over all $i$ (ergodicity), or elsewhere you must repeat your experiment several times, getting several $d_i$ and estimating $m_i$ by averaging for each $i$. $\endgroup$ – leonbloy Apr 22 '12 at 15:40
  • $\begingroup$ @leonbloy I've expanded the question another bit, I'm still not sure how should I calculate those $m_i$ values. $\endgroup$ – Gabriel Apr 22 '12 at 23:55
6
$\begingroup$

I think that the sentence is poorly worded, one should not use the word "expect" in that context, it can induce confusion. Because to "expect either 0 or 1 count per bin" does NOT mean that "the expected value per bin is either 0 or 1 ($m_i=0$ or $m_i=1$)". It rather means that the expected value per bin is quite small ($m_i$ is quite below 1 - but not zero!), so that the probability of $d_i$ being 0 or 1 is high (or put in other way, the likelihood of the counting being greater than 1 is small).

For example, if $m_i=0.2$, we'd have a Poisson with probabilites $L_0 = 0.8187$, $L_1=0.1638$, so the probability that we have "either 0 or 1 count per bin" is high (0.9825); and that's what the (confusing) expression "we expect either 0 or 1 count per bin" means.

$\endgroup$
  • $\begingroup$ I've explained the issue a bit more detailed @leonbloy, I'd really appreciate any further comments. $\endgroup$ – Gabriel Apr 22 '12 at 15:28
  • 1
    $\begingroup$ See my comment. I emphasize: $m_i$ is not a random variable, it's not something that "takes values" (in each realization), but a parameter that is fixed, and that you just estimate. What "takes values", the random variable (number of values that fall in each bine), is $d_i$. $\endgroup$ – leonbloy Apr 22 '12 at 16:13
3
$\begingroup$

If $m_i=0$ then the likelihood $L_i$ of observing $0$ is $1$ and of observing anything else is $0$.

This is an example of the wider convention where $0^0$ is taken to be $1$ so $\frac{0^0}{0!}\exp(-0)=1$

$\endgroup$
  • $\begingroup$ Your answer seems to contradict leonbloy's answer given earlier @Henry. I've explained this better in the original question. $\endgroup$ – Gabriel Apr 22 '12 at 15:30
1
$\begingroup$

The statement "Suppose that in a particular model $m$, you expect there to be $m_i$ counts in bin $i$. Then if the model is correct the likelihood of actually observing $d_i$ counts in bin $i$ of the data is, from the Poisson distribution,*

$L_i = m_i^{d_i}/d_i! \; exp(-m_i)$"

is incorrect in a couple of ways. First, $L$ is not the formula for the Poisson distribution. The Poisson has one parameter, let us label it $\lambda$, not one parameter $m_i$ for each bin $i$. The parameter $\lambda$ is the expected value of a random variable from the distribution, not the expected value of the number of counts in a specific bin. Also, the use of $d_i$ is confusing. Here's the correct probability for a single draw from a Poisson to equal $i$:

$L_i = \frac{\lambda^{i}}{i!} \exp(-\lambda)$

Note that $L_i > 0 \space \forall i \ge 0$. If we make $N$ draws from a Poisson, $NL_i$ can be interpreted as the expected number of draws which will have value $i$. This is the same as $m_i$ in the initial statement. As $L_i > 0$, $m_i > 0$ as well. $m_i$ can be any positive real number, as in the initial statement.

The initial statement would seem to lead more naturally to a multinomial distribution, where the expected number of counts in a bin divided by the total number of counts is indeed a parameter, and there is one such for each bin. When reduced to just "counts in bin $i$ vs. counts not in bin $i$" the multinomial simplifies to a binomial; the probability of seeing $d_i$ counts in bin $i$ given $N$ total counts and an expected number of counts in bin $i$ of $m_i$ is:

$L_i = \binom {N}{d_i}(m_i/N)^{d_i}(1-m_i/N)^{N-d_i}$

I'm a little hesitant to post this, because one doesn't expect lecture notes to be this wrong, which implies a significant chance that my brain has malfunctioned and I've just missed something myself.

$\endgroup$
  • $\begingroup$ jbowman Please see Edit 2 where I cite a paper where a very similar analysis is performed. What do you think of that analysis? Is that correct? Do you have any idea how the authors calculated $\nu_i(\theta)$? $\endgroup$ – Gabriel Apr 22 '12 at 23:57
  • $\begingroup$ What we have in Edit 2 is a case where you are doing Poisson regression - you have some observations indexed by $i$ and some other relevant variables that may be different across observations, e.g., male/female, age category, ... so each observation might have a different Poisson mean parameter. The mean parameter for observation $i$ is denoted $v_i$, and is in turn a function of parameters $\theta$. To emphasize: the observations are drawn from different Poisson distributions. This is a special case of a generalized linear model, and estimation can be done using glm software. $\endgroup$ – jbowman Apr 23 '12 at 0:16
  • $\begingroup$ The index $i$ in the analysis shown in Edit 2 runs through the bins in each modeled and observed histograms (both histograms have obviously the same number of bins) and the parameter $\theta$ is what I can change to obtain different model histograms. What I'm not understanding is how to calculate that $\nu_i(\theta)$ parameter. $\endgroup$ – Gabriel Apr 23 '12 at 0:28
1
$\begingroup$

I think that, you are using Bayesian statistics here. The posterior probability is proportional to likelihood function, then you have

P(model(parameters)|data) \propto L(data|model(parameters)).

Therefore, you can have such problem. Actually, I have a similar problem. Nonetheless, I use greater bins. It is a possible solution, to avoid the problem for bins with zero value in the model.

$\endgroup$
  • 1
    $\begingroup$ If you truly have a zero expectation in a bin and there is an observed event there, then you have definitively demonstrated the incorrectness of your model. Combining bins only papers over this fundamental problem--it certainly doesn't solve it! $\endgroup$ – whuber Apr 11 '17 at 17:53
  • 1
    $\begingroup$ What in the question indicates it involves Bayesian methods? $\endgroup$ – Michael Chernick Apr 11 '17 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.