4
$\begingroup$

I have observed a number ($n=6$) of events and want to test the null hypothesis that they are Poisson distributed with a known parameter $\lambda\approx1$. But I don't know exactly when I started and stopped observing, only when my first and last observation are. How can I test this hypothesis while avoiding the bias of using the first and last events explicitly?

Background

This question comes from probabilistic number theory. It's common to treat the prime divisors $p$ of a typical (big) number as random variables which are present with probability $1/p$ and absent otherwise. At large scales, the presence of a prime in the range $[x, y]$ can be modeled as a Poisson process with $\lambda=\log\log y-\log\log x.$

My question, then, is to decide whether a particular number is 'typical' or not: are the primes distributed in such an unusual fashion that it would be unreasonable to conclude that it was chosen according to this random process?

In fact I have a competing model, but I have not formalized it. The number may instead have an algebraic factorization, where it would have a large number of factors in a similar size range. I don't want to model this directly, merely to test the better-understood Poisson model above.

I have a (partial) list of the prime factors of this number, $p_1\le p_2\le p_3\le\cdots\le p_k$. Because the unfactored part of the number is large and there are probably no undiscovered primes between the ones I have already found, there should be no need to modify the Poisson model. (If I had the full factorization then there would be a degrees-of-freedom issue that the product of the primes would need to be equal to the original number.) So basically I want to check if there are roughly the expected number of primes between $p_1$ and $p_k$.

But this isn't as easy as checking if $$ \frac{\lambda^ke^{-\lambda}}{k!} $$ with $\lambda=\log\log p_k-\log\log p_1$ is smaller than some $\alpha,$ because then I would have picked $p_1$ and $p_k$ based on the factors I already knew I would have. How can I compensate for this bias?

Sidenote: I don't want to model selection bias in terms of 'did you only look at this number because you observed an unusual distribution of factors, and how many other numbers did you look at'. In fact I suspected that this number might have these unusual features even before I factored it.

Resolution

In case anyone's interested, I took the number to some people knowledgeable about these sorts of things and, indeed, it has an algebraic factorization and so it was correct to reject the null hypothesis. Thanks!

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ And you don't know when the intermediate observations are? If you do, then the successive differences will comprise 5 independent exponential random variables. $\endgroup$ – Russ Lenth Jul 27 '14 at 20:16
  • $\begingroup$ @RussLenth: I do have the intermediate observations. In terms of $\lambda$ they are 0.318388897, 0.0893040538, 0.209697071, 0.0782522406, and 0.265018191. $\endgroup$ – Charles Jul 27 '14 at 20:25
  • $\begingroup$ I should mention that it's possible that there's another observation in between two of these, but (1) that's unlikely and (2) it would only make $H_0$ less likely. $\endgroup$ – Charles Jul 27 '14 at 20:26
5
$\begingroup$

[Caution: Not understanding the background very well, so am taking your word for it that the Poisson model with $\lambda=1$ is the appropriate thing to test.]

Given the data you provided for the time differences, the sum of them should have a gamma distribution with shape parameter $\alpha=5$ and scale parameter $\beta=1$ if the null hypothesis that $\lambda=1$ is true.

I have these results for the two-sided test at a .05 significance level.

> qgamma(c(.025,.975), 5, 1)
[1]  1.623486 10.241589

> sum(0.318388897, 0.0893040538, 0.209697071, 0.0782522406, 0.265018191)
[1] 0.9606605

Your observed sum is less than the lower quantile, so the null is rejected. It appears that $\beta<1$, i.e., $\lambda > 1$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ If I understand you correctly: I should reject $H_0$ (at 0.05 significance) because $\alpha=\frac{1}{\Gamma(5)1^5}\int_0^{0.9606605}x^4e^{-x}\approx0.0031<0.05$. Right? $\endgroup$ – Charles Jul 27 '14 at 21:24
  • 3
    $\begingroup$ @Charles - Sorta right. I think the P value is $2\times.0031 = .0062$, because you did not state a priori which direction $\lambda$ should deviate from $1$. I did state a one-tailed conclusion, though. In rejecting a two-tailed test, you may -- and should -- state a one-tailed conclusion in the appropriate direction. $\endgroup$ – Russ Lenth Jul 27 '14 at 23:55
  • $\begingroup$ Russ: FYI your conclusion was correct in my case -- the number did have an algebraic factorization (i.e., didn't follow the proposed model). Thanks! $\endgroup$ – Charles Aug 5 '14 at 19:15
  • $\begingroup$ The number of events seems a bit small to be using Poisson, e.g., see link. $\endgroup$ – Carl Nov 12 '16 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.