Does anyone know of a name for the following statistical test/whether there is a name/whether the following line of reasoning is just bogus?

Say I have a value $x$, and a way to generate samples from a null distribution, and I want to test the null hypothesis that $x$ was sampled from this distribution. So I generate samples one at a time and stop on the first sample that's bigger than my test value $x$. Call this the $n^{th}$ sample.

Say the null distribution has cdf $F(x)$. If $x$ came from this distribution, then the probability that the first larger sample is the $n^{th}$ sample can be written

$$\int dx F'(x) F(x)^n \left(1- F(x) \right) $$

This is just a weighted sum of geometric distributions for the "first bigger than" time, weighted by the probability of setting the threshold $x$ on the first sample.

This integral is just $\frac{1}{n} - \frac{1}{n+1}$. (Another way to derive this result is just to note that if $n+1$ samples are drawn, for the last sample to be the first one larger than $x$, the largest value in the group must be the last sample, the second largest value must be $x$, and any other ordering of the remaining $n-1$ is okay. So the probability of a correct ordering is $\frac{(n-1)!}{(n+1)!} = \frac{1}{n} - \frac{1}{n+1}$).

The probability of getting a "first bigger than" time of $\frac{1}{N}$ or greater is then

$$\sum_{n=N}^\infty \frac{1}{n} - \frac{1}{n+1} = \frac{1}{N} $$

So $\frac{1}{N}$ can be used as a p-value for the null hypothesis that the sample $x$ came from the distribution $F$.

One reason for using a test like this (ie. the reason I want to use it) is because I don't know the cdf $F$ and it's also relatively costly to generate samples from it, so I'd rather not send it off running 1 million samples in order to have the ability to generate low p-values, only to find that 10 out of the first 50 samples were bigger than $x$.

Side note: anyone know what this pmf $p(n) = \frac{1}{n} - \frac{1}{n+1}$ is called? It's a bit weird - it's well-normalized but has an infinite first moment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.