29
$\begingroup$

I have generated a vector which has a Poisson distribution, as follows:

x = rpois(1000,10)

If I make a histogram using hist(x), the distribution looks like a the familiar bell-shaped normal distribution. However, a the Kolmogorov-Smirnoff test using ks.test(x, 'pnorm',10,3) says the distribution is significantly different to a normal distribution, due to very small p value.

So my question is: how does the Poisson distribution differ from a normal distribution, when the histogram looks so similar to a normal distribution?

$\endgroup$
  • $\begingroup$ Also (as an add-in to David's answer): read this (stats.stackexchange.com/a/2498/603) and set your sample size to 100 and see the difference it makes. $\endgroup$ – user603 Jul 16 '12 at 19:28
20
$\begingroup$
  1. A Poisson distribution is discrete while a normal distribution is continuous, and a Poisson random variable is always >= 0. Thus, a Kolgomorov-Smirnov test will often be able to tell the difference.

  2. When the mean of a Poisson distribution is large, it becomes similar to a normal distribution. However, rpois(1000, 10) doesn't even look that similar to a normal distribution (it stops short at 0 and the right tail is too long).

  3. Why are you comparing it to ks.test(..., 'pnorm', 10, 3) rather than ks.test(..., 'pnorm', 10, sqrt(10))? The difference between 3 and $\sqrt{10}$ is small but will itself make a difference when comparing distributions. Even if the distribution truly were normal you would end up with an anti-conservative p-value distribution:

    set.seed(1)
    
    hist(replicate(10000, ks.test(rnorm(1000, 10, sqrt(10)), 'pnorm', 10, 3)$p.value))
    

enter image description here

$\endgroup$
  • 3
    $\begingroup$ Often people will see something vaguely symmetric and assume it looks "normal." I suspect that what @Ross saw. $\endgroup$ – Fraijo Jul 16 '12 at 19:31
  • 2
    $\begingroup$ Note that the KS test generally assumes continuous distributions, so relying on the reported p-value in this case may (also) be somewhat suspect. $\endgroup$ – cardinal Jul 16 '12 at 19:55
  • 1
    $\begingroup$ True: running hist(replicate(1000, ks.test(rpois(1000, 10), rpois(1000, 10))$p.value)) demonstrates that a test comparing two identical Poisson distributions would be too conservative. $\endgroup$ – David Robinson Jul 16 '12 at 19:58
  • $\begingroup$ @Fraijo: indeed. We have a more general question on this theme: If my histogram shows a bell-shaped curve, can I say my data is normally distributed? $\endgroup$ – Silverfish Jan 17 '15 at 23:03
17
$\begingroup$

Here's much easier way to understand it:

You can look at Binomial distribution as the "mother" of most distributions. The normal distribution is just an approximation of Binomial distribution when n becomes large enough. In fact, Abraham de Moivre essentially discovered normal distribution while trying to approximate Binomial distribution because it quickly goes out of hand to compute Binomial distribution as n grows especially when you don't have computers (reference).

Poisson distribution is also just another approximation of Binomial distribution but it holds much better than normal distribution when n is large and p is small, or more precisely when average is approximately same as variance (remember that for Binomial distribution, average = np and var = np(1-p)) (reference). Why is this particular situation so important? Apparently it surfaces a lot in real world and that's why we have this "special" approximation. Below example illustrates scenarios where Poisson approximation works really great.

Example

We have a datacenter of 100,000 computers. Probability of any given computer failing today is 0.001. So on average np=100 computers fail in data center. What is the probability that only 50 computers will fail today?

Binomial: 1.208E-8
Poisson: 1.223E-8
Normal: 1.469E-7

In fact, the approximation quality for normal distribution goes down the drain as we go in the tail of the distribution but Poisson continues to holds very nicely. In above example, let's consider what is the probability that only 5 computers will fail today?

Binomial: 2.96E-36 
Poisson: 3.1E-36
Normal: 9.6E-22

Hopefully, this gives you better intuitive understanding of these 3 distributions.

$\endgroup$
  • $\begingroup$ What an amazing and great answer! Thanks a lot. :) $\endgroup$ – Bora M. Alper Nov 20 '16 at 17:59
11
$\begingroup$

I think it is worth mentioning that a Poisson($\lambda$) pmf is the limiting pmf of a Binomial($n$,$p_n$) with $p_n = \lambda / n$.

One rather lengthy development can be found on this blog.

But, we can prove this economically here as well. If $X_n \sim \mathrm{Binomial}(n,\lambda/n)$ then for fixed $k$ $$ \begin{align} \mathbb P(X_n = k) &= \frac{n!}{k!(n-k)!} \left(\frac{\lambda}{n}\right)^k \left(1-\frac{\lambda}{n}\right)^{n-k} \\ &= \underbrace{\frac{n! n^{-k}}{(n-k)!}}_{\to 1} \frac{\lambda^k}{k!}\underbrace{(1-\lambda/n)^n}_{\to e^{-\lambda}} \cdot \underbrace{(1-\lambda/n)^{-k}}_{\to 1} \>. \end{align} $$

The first and last terms are easily seen to converge to 1 as $n \to \infty$ (recalling that $k$ is fixed). So, $$ \mathbb P(X_n = k) \to \frac{e^{-\lambda} \lambda^k}{k!} \,, $$ as $n \to \infty$ since $(1-\lambda/n)^n \to e^{-\lambda}$.

In addition one has the normal approximation to the Binomial, i.e., Binomial($n$,$p$) $\approxeq^d \mathcal N(np, np(1-p))$. The approximation improves as $n \rightarrow \infty$ and $p$ stays away from 0 and 1. Obviously for the Poisson regime this is not the case (since there $p_n = \lambda / n \rightarrow 0$) but the larger $\lambda$ is the larger $n$ can be and still have a reasonable normal approximation.

$\endgroup$
  • $\begingroup$ (+1) Welcome to the site. I've made a few edits; please check that I have not introduced any errors in the process. I was not quite sure of what to make of the very last phrase in the last sentence. Some additional clarification there might be helpful. $\endgroup$ – cardinal Aug 17 '12 at 0:24
  • 1
    $\begingroup$ I like the direction of this, though there may be ways to relate it a little more closely to the question at hand by making the connections between the three distributions clearer. For example (a) A binomial random variable (sequence) acts like a Poisson as long as $n p_n \approx \lambda$, (b) A binomial (sequence) acts like a normal as long as $p$ is approximately a fixed constant and (c) a Poisson (sequence) acts like a normal for large $\lambda$ essentially due to its infinite divisibility. $\endgroup$ – cardinal Aug 17 '12 at 1:56
  • 1
    $\begingroup$ Nice comments @cardinal. About the last sentence, for fixed, large $n$ the larger $\lambda$ the larger $p_n$ (e.g. closer to $1/2$). Therefore the better the Normal approximation to the Binomial and in turn the Poisson. $\endgroup$ – muratoa Aug 17 '12 at 5:30
  • $\begingroup$ Thanks. I see what you were trying to say now. I generally agree, with the caveat that some care needs to be taken with the relationship between the parameters, which are considered fixed and which vary with the others. :) $\endgroup$ – cardinal Aug 17 '12 at 12:29
  • $\begingroup$ Hi Murat and welcome to the site! it's good to see you here and I hope you stick around. +1 for explaining why the histogram of a poisson looks a lot like that of a normal when $\lambda$ is large. $\endgroup$ – Macro Aug 17 '12 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.