Skip to main content
edited tags
Link
Tim
  • 141.1k
  • 26
  • 270
  • 512
Source Link
jed
  • 3
  • 1

Poisson goodness-of-fit of small sample

Please note that I have no background in statistics - this is for personal interest.

The post is inspired by a question at work.

We were given data by another department asserting that for a continuously advertised position there were zero applications in 6 consecutive days, followed by 7 applications on the 7th day. The validity of the data was questioned.

I wondered what the probability was of getting this exact result on seven consecutive days, assuming a homogeneous Poisson distribution. The below may or may not be correct.

$P(X=k) = \frac{{e^{-\lambda}}\lambda^k}{k!}$

So,

$P(X=0) = {\frac{{e^{-\lambda}}\lambda^0}{0!}}={e^{-\lambda}}$

$P(X=7) = {\frac{{e^{-\lambda}}\lambda^7}{7!}} = {\frac{{e^{-\lambda}}\lambda^7}{5040}}$

$P(X=\{0,0,0,0,0,0,7\}) = \frac{{(e^{-\lambda})}^6{e^{-\lambda}}\lambda^7}{5040} = \frac{e^{-7\lambda}\lambda^7}{5040}$

Finding the value of $\lambda$ that gives the maximum likelihood for $P(X=\{0,0,0,0,0,0,7\})$,

${\frac{\partial}{\partial\lambda}}\frac{e^{-7\lambda}\lambda^7}{5040}$ $ = {\frac{1}{5040}}\frac{\partial}{\partial\lambda}{e^{-7\lambda}}\lambda^7$

Using the product rule,

${\frac{d}{dx}}(uv) = v{\frac{du}{dx}} + u{\frac{dv}{dx}}$ , where $u = {e^{-7\lambda}}$, $v = {\lambda^7}$, $x = \lambda$ gives:

${\frac{d}{d\lambda}}({e^{-7\lambda}}{\lambda^7}) = {\lambda^7}{\frac{d}{d\lambda}}{e^{-7\lambda}} + {e^{-7\lambda}}{\frac{d}{d\lambda}}{\lambda^7}$

Using the chain rule,

${\frac{dg}{dx}} = {\frac{dg}{du}}{\frac{du}{dx}}$ , where $g = e^{u}$, $x = \lambda$, $u = -7\lambda$ gives:

${\frac{d}{d\lambda}}e^{-7\lambda} = {\frac{d}{du}}e^u{\frac{d}{d\lambda}}-7\lambda$ $ = {-7e^{-7\lambda}}$

So,

${\frac{d}{d\lambda}}({e^{-7\lambda}}{\lambda^7}) = {-7e^{-7\lambda}}{\lambda^7} + {7e^{-7\lambda}}{\lambda^6}$ $ = {7e^{-7\lambda}}{\lambda^6}(1 - \lambda) = 0$

$\lambda = \{0, 1\}$

So, with a maximum probability at $\lambda = 1$,

$P(X=\{0,0,0,0,0,0,7\}) = \frac{e^{-7}1^7}{5040} = \frac{1}{5040e^7} \approx \frac{1}{5527031}$

This seemed like pretty slim odds, but on reflection, it wasn't telling me much. Out of an infinite number of possible outcomes, the chosen outcome had a 1 in 5.5 million chance of occurring.

My question - is there a good test to use to give an indication of how well this data fit a homogeneous Poisson process? What would the results of such as test actually tell me?