Poisson goodness-of-fit of small sample

Question

Please note that I have no background in statistics - this is for personal interest.

The post is inspired by a question at work.

We were given data by another department asserting that for a continuously advertised position there were zero applications in 6 consecutive days, followed by 7 applications on the 7^th day. The validity of the data was questioned.

I wondered what the probability was of getting this exact result on seven consecutive days, assuming a homogeneous Poisson distribution. The below may or may not be correct.

$P(X=k) = \frac{{e^{-\lambda}}\lambda^k}{k!}$

So,

$P(X=0) = {\frac{{e^{-\lambda}}\lambda^0}{0!}}={e^{-\lambda}}$

$P(X=7) = {\frac{{e^{-\lambda}}\lambda^7}{7!}} = {\frac{{e^{-\lambda}}\lambda^7}{5040}}$

$P(X=\{0,0,0,0,0,0,7\}) = \frac{{(e^{-\lambda})}^6{e^{-\lambda}}\lambda^7}{5040} = \frac{e^{-7\lambda}\lambda^7}{5040}$

Finding the value of $\lambda$ that gives the maximum likelihood for $P(X=\{0,0,0,0,0,0,7\})$,

${\frac{\partial}{\partial\lambda}}\frac{e^{-7\lambda}\lambda^7}{5040}$ $ = {\frac{1}{5040}}\frac{\partial}{\partial\lambda}{e^{-7\lambda}}\lambda^7$

Using the product rule,

${\frac{d}{dx}}(uv) = v{\frac{du}{dx}} + u{\frac{dv}{dx}}$ , where $u = {e^{-7\lambda}}$, $v = {\lambda^7}$, $x = \lambda$ gives:

${\frac{d}{d\lambda}}({e^{-7\lambda}}{\lambda^7}) = {\lambda^7}{\frac{d}{d\lambda}}{e^{-7\lambda}} + {e^{-7\lambda}}{\frac{d}{d\lambda}}{\lambda^7}$

Using the chain rule,

${\frac{dg}{dx}} = {\frac{dg}{du}}{\frac{du}{dx}}$ , where $g = e^{u}$, $x = \lambda$, $u = -7\lambda$ gives:

${\frac{d}{d\lambda}}e^{-7\lambda} = {\frac{d}{du}}e^u{\frac{d}{d\lambda}}-7\lambda$ $ = {-7e^{-7\lambda}}$

So,

${\frac{d}{d\lambda}}({e^{-7\lambda}}{\lambda^7}) = {-7e^{-7\lambda}}{\lambda^7} + {7e^{-7\lambda}}{\lambda^6}$ $ = {7e^{-7\lambda}}{\lambda^6}(1 - \lambda) = 0$

$\lambda = \{0, 1\}$

So, with a maximum probability at $\lambda = 1$,

$P(X=\{0,0,0,0,0,0,7\}) = \frac{e^{-7}1^7}{5040} = \frac{1}{5040e^7} \approx \frac{1}{5527031}$

This seemed like pretty slim odds, but on reflection, it wasn't telling me much. Out of an infinite number of possible outcomes, the chosen outcome had a 1 in 5.5 million chance of occurring.

My question - is there a good test to use to give an indication of how well this data fit a homogeneous Poisson process? What would the results of such as test actually tell me?

Answer 1

Good efforts and nice thinking. Just a few notes about your method and your conclusion.

Estimating $\lambda$ could have been done by using the mean which is also an estimator of lambda. $7applications / 7days=1$

This is indeed pretty slim, but some data more "usual" could get pretty slim odds too. The probability of a specific event is rarely the good approach to test an hypothesis. The event [0,2,1,1,1,2] would have a very very small probability too, but it does not appear as abnormal.

The best would have been to consider the range of "what could have been seen as very weird data" and compute the probability to get an event in this whole event list. But how to build this dataset is a tricky question.

One could also point out some real world effect there. On what basis can we say that this should follow a poisson law ? I am personally a late applicant, maybe many others are. Under this track, what you could do is to check if poisson law is a good fit for this data. Maybe this is what you wanted to do since the beginning. You could then continue your self-study journey by looking at how to do a goodness of fit test to check that ! I think it would be rejected. Edit : I reread your question, this is what you are specifically asking, I hope other comments would interest you anyway.

A last comment, more qualitative, is that by estimating the best $\lambda$ fitting your data, you overestimate the probability of these datas. If lambda was in reality a bit less or more, the probability of your single event would go down. This is (very very broadly) the idea of degree of freedom that you may encounter while doing a goodness of fit test.