How is Poisson distribution different to normal distribution?

Question

I have generated a vector which has a Poisson distribution, as follows:

x = rpois(1000,10)

If I make a histogram using hist(x), the distribution looks like a the familiar bell-shaped normal distribution. However, a the Kolmogorov-Smirnoff test using ks.test(x, 'pnorm',10,3) says the distribution is significantly different to a normal distribution, due to very small p value.

So my question is: how does the Poisson distribution differ from a normal distribution, when the histogram looks so similar to a normal distribution?

Answer 1

1. A Poisson distribution is discrete while a normal distribution is continuous, and a Poisson random variable is always >= 0. Thus, a Kolgomorov-Smirnov test will often be able to tell the difference.

2. When the mean of a Poisson distribution is large, it becomes similar to a normal distribution. However, rpois(1000, 10) doesn't even look that similar to a normal distribution (it stops short at 0 and the right tail is too long).

3. Why are you comparing it to ks.test(..., 'pnorm', 10, 3) rather than ks.test(..., 'pnorm', 10, sqrt(10))? The difference between 3 and $\sqrt{10}$ is small but will itself make a difference when comparing distributions. Even if the distribution truly were normal you would end up with an anti-conservative p-value distribution:

   set.seed(1)
   
   hist(replicate(10000, ks.test(rnorm(1000, 10, sqrt(10)), 'pnorm', 10, 3)$p.value))
   

Answer 2

Here's much easier way to understand it:

You can look at Binomial distribution as the "mother" of most distributions. The normal distribution is just an approximation of Binomial distribution when n becomes large enough. In fact, Abraham de Moivre essentially discovered normal distribution while trying to approximate Binomial distribution because it quickly goes out of hand to compute Binomial distribution as n grows especially when you don't have computers (reference).

Poisson distribution is also just another approximation of Binomial distribution but it holds much better than normal distribution when n is large and p is small, or more precisely when average is approximately same as variance (remember that for Binomial distribution, average = np and var = np(1-p)) (reference). Why is this particular situation so important? Apparently it surfaces a lot in real world and that's why we have this "special" approximation. Below example illustrates scenarios where Poisson approximation works really great.

Example

We have a datacenter of 100,000 computers. Probability of any given computer failing today is 0.001. So on average np=100 computers fail in data center. What is the probability that only 50 computers will fail today?

Binomial: 1.208E-8
Poisson: 1.223E-8
Normal: 1.469E-7

In fact, the approximation quality for normal distribution goes down the drain as we go in the tail of the distribution but Poisson continues to holds very nicely. In above example, let's consider what is the probability that only 5 computers will fail today?

Binomial: 2.96E-36 
Poisson: 3.1E-36
Normal: 9.6E-22

Hopefully, this gives you better intuitive understanding of these 3 distributions.

Answer 3

I think it is worth mentioning that a Poisson($\lambda$) pmf is the limiting pmf of a Binomial($n$,$p_n$) with $p_n = \lambda / n$.

One rather lengthy development can be found on this blog.

But, we can prove this economically here as well. If $X_n \sim \mathrm{Binomial}(n,\lambda/n)$ then for fixed $k$ $$ \begin{align} \mathbb P(X_n = k) &= \frac{n!}{k!(n-k)!} \left(\frac{\lambda}{n}\right)^k \left(1-\frac{\lambda}{n}\right)^{n-k} \\ &= \underbrace{\frac{n! n^{-k}}{(n-k)!}}_{\to 1} \frac{\lambda^k}{k!}\underbrace{(1-\lambda/n)^n}_{\to e^{-\lambda}} \cdot \underbrace{(1-\lambda/n)^{-k}}_{\to 1} \>. \end{align} $$

The first and last terms are easily seen to converge to 1 as $n \to \infty$ (recalling that $k$ is fixed). So, $$ \mathbb P(X_n = k) \to \frac{e^{-\lambda} \lambda^k}{k!} \,, $$ as $n \to \infty$ since $(1-\lambda/n)^n \to e^{-\lambda}$.

In addition one has the normal approximation to the Binomial, i.e., Binomial($n$,$p$) $\approxeq^d \mathcal N(np, np(1-p))$. The approximation improves as $n \rightarrow \infty$ and $p$ stays away from 0 and 1. Obviously for the Poisson regime this is not the case (since there $p_n = \lambda / n \rightarrow 0$) but the larger $\lambda$ is the larger $n$ can be and still have a reasonable normal approximation.

Answer 4

It's a great question because Poisson distribution is not only different, but it is also so similar to Normal distribution. Here's how it is similar:

• the sum of two normals is normal, so is the sum of two Poissons
• Brownian motion (Gaussian) and Poisson process are both Levy processes
• Both Poisson and Gaussian distributions can be approximations of the Binomial distribution with large N
• for large $\lambda$ Gaussian distribution looks very much like Poisson, which you already noticed