3
$\begingroup$

Does anyone know of a name for the following statistical test/whether there is a name/whether the following line of reasoning is just bogus?

Say I have a value $x$, and a way to generate samples from a null distribution, and I want to test the null hypothesis that $x$ was sampled from this distribution. So I generate samples one at a time and stop on the first sample that's bigger than my test value $x$. Call this the $n^{th}$ sample.

Say the null distribution has cdf $F(x)$. If $x$ came from this distribution, then the probability that the first larger sample is the $n^{th}$ sample can be written

$$\int dx F'(x) F(x)^n \left(1- F(x) \right) $$

This is just a weighted sum of geometric distributions for the "first bigger than" time, weighted by the probability of setting the threshold $x$ on the first sample.

This integral is just $\frac{1}{n} - \frac{1}{n+1}$. (Another way to derive this result is just to note that if $n+1$ samples are drawn, for the last sample to be the first one larger than $x$, the largest value in the group must be the last sample, the second largest value must be $x$, and any other ordering of the remaining $n-1$ is okay. So the probability of a correct ordering is $\frac{(n-1)!}{(n+1)!} = \frac{1}{n} - \frac{1}{n+1}$).

The probability of getting a "first bigger than" time of $\frac{1}{N}$ or greater is then

$$\sum_{n=N}^\infty \frac{1}{n} - \frac{1}{n+1} = \frac{1}{N} $$

So $\frac{1}{N}$ can be used as a p-value for the null hypothesis that the sample $x$ came from the distribution $F$.

One reason for using a test like this (ie. the reason I want to use it) is because I don't know the cdf $F$ and it's also relatively costly to generate samples from it, so I'd rather not send it off running 1 million samples in order to have the ability to generate low p-values, only to find that 10 out of the first 50 samples were bigger than $x$.

Side note: anyone know what this pmf $p(n) = \frac{1}{n} - \frac{1}{n+1}$ is called? It's a bit weird - it's well-normalized but has an infinite first moment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.