Poisson distribution and statistical significance

Question

Lets say I have a website which gets 100 hits per day (mu = 100). Yesterday my website got 130 hits (x = 130). If I assume a Poisson distribution, then the probability of getting 130 hits is:

> dpois(130, 100)
[1] 0.0005752527 # about 0.06%

So this tells me that getting 130 hits is quite unusual for my website due to the low probability.

My understanding of statistical significance is that it is used to determine whether the outcome of an experiment is due either to chance or some kind of deterministic relationship.

1. How would I apply that in this situation?
2. What test should one use? (and is it in R?)

Many thanks in advance for your time.

Note: I saw someone at a business talk asked something very similar to this and I had no idea what they meant by it, and so now I'm just trying to educate myself. I'm new to R, but that seems like the software most used for these kind of questions, hence my request.

Answer 1

There are two points to make:

1. It is not the specific value of 130 that is unusual, but that it is much larger than 100. If you got more than 130 hits, that would have been even more surprising. So we usually look at the P(X>=130), not just P(X=130). By your logic even 100 hits would be unusual, because dpois(100,100)=0.04. So a more correct calculation is to look at ppois(129, 100, lower=F)=0.00228. This is still small, but not as extreme as your value. And this does not even take into account, that an unusually low number of hits might also surprise you. We often multiply the probability of exceeding the observed count by 2 to account for this.
2. If you keep checking your hits every day, sooner or later even rare events will occur. For example P(X>=130) happens to be close to 1/365, so such an event would be expected to occur once a year.

Answer 2

First, note that dpois(130, 100) will give you the probability of exactly 130 hits if you are assuming that the true rate is 100. That probability is indeed very low. However, in the usual hypothesis testing framework, what we calculate is the probability of the observed outcome or an even more extreme outcome. You can obtain this for the Poisson distribution with:

> ppois(129, lambda=100, lower.tail=FALSE)
[1] 0.002282093

So, there is a ~.2% probability of observing the 130 hits or even more hits if you are assuming a true rate of 100. By convention, if this value is below .025 (which it is), we would consider this finding "statistically significant" at $\alpha = .05$ (two-sided). What this means is that you are willing to take a 5% risk that your decision (calling the deviation statistically significant and rejecting the hypothesis that the true rate is 100 for that observation) is wrong. That is, if the true rate is indeed 100 for that day, then in 2.5% of the cases, the observed rate will in fact be 120 or larger (qpois(.975, lambda=100)) and in 2.5% of the cases, the observed rate will be 81 or lower (qpois(.025, lambda=100)). So, if you are using $\alpha = .05$, then in 5% of the cases, your decision will be false.