# Estimating the probability in a Bernoulli process by sampling until 10 failures: is it biased?

Suppose we have a Bernoulli process with failure probability $q$ (which will be small, say, $q \leq 0.01$) from which we sample until we encounter $10$ failures. We thereby estimate the probability of failure as $\hat{q}:=10/N$ where $N$ is the number of samples.

Question: Is $\hat{q}$ a biased estimate of $q$? And, if so, is there a way to correct it?

I'm concerned that insisting the last sample is a fail biases the estimate.

It is true that $\hat{q}$ is a biased estimate of $q$ in the sense that $\text{E}(\hat{q}) \neq q$, but you shouldn't necessarily let this deter you. This exact scenario can be used as a criticism against the idea that we should always use unbiased estimators, because here the bias is more of an artifact of the particular experiment we happen to be doing. The data look exactly as they would if we had chosen the number of samples in advance, so why should our inferences change?

Interestingly, if you were to collect data in this way and then write down the likelihood function under both the binomial (fixed sample size) and negative binomial models, you would find that the two are proportional to one another. This means that $\hat{q}$ is just the ordinary maximum likelihood estimate under the negative binomial model, which of course is a perfectly reasonable estimate.

• Great! It looks like (for my purposes) this bias is not an issue. – becky Mar 7 '16 at 8:38

It is not insisting that the last sample is a fail which biases the estimate, it is taking the reciprocal of $N$

So $\mathbb{E}\left[\frac{N}{10}\right] =\frac{1}{q}$ in your example but $\mathbb{E}\left[\frac{10}{N}\right] \not = q$. This is close to comparing the arithmetic mean with the harmonic mean

The bad news is that the bias can increase as $q$ gets smaller, though not by much once $q$ is already small. The good news is that bias decreases as the required number of failures increases. It seems that if you require $f$ failures, then the bias is bounded above by a multiplicative factor of $\frac{f}{f-1}$ for small $q$; you do not want this approach when you stop after the first failure

Stopping after $10$ failures, with $q=0.01$ you will get $\mathbb{E}\left[\frac{N}{10}\right] = 100$ but $\mathbb{E}\left[\frac{10}{N}\right] \approx 0.011097$, while with $q=0.001$ you will get $\mathbb{E}\left[\frac{N}{10}\right] = 1000$ but $\mathbb{E}\left[\frac{10}{N}\right] \approx 0.001111$. A bias of roughly a $\frac{10}{9}$ multiplicative factor

As a complement to dsaxton's answer, here are some simulations in R showing the sampling distribution of $\hat{q}$ when $k=10$ and $q_0 = 0.02$:

n_replications <- 10000
k <- 10
failure_prob <- 0.02
n_trials <- k + rnbinom(n_replications, size=k, prob=failure_prob)
all(n_trials >= k)  # Sanity check, cannot have 10 failures in < 10 trials

estimated_failure_probability <- k / n_trials
histogram_breaks <- seq(0, max(estimated_failure_probability) + 0.001, 0.001)
## png("estimated_failure_probability.png")
hist(estimated_failure_probability, breaks=histogram_breaks)
abline(v=failure_prob, col="red", lty=2, lwd=2)  # True failure probability in red
## dev.off()

mean(estimated_failure_probability)  # Around 0.022
sd(estimated_failure_probability)
t.test(x=estimated_failure_probability, mu=failure_prob)  # Interval around [0.0220, 0.0223]


It looks like $\mathbb{E}\left[ \hat{q}\right] \approx 0.022$, which is a rather small bias relative to the variability in $\hat{q}$. • That's really helpful. At that level, it's not worth me worrying about. – becky Mar 7 '16 at 8:40
• You can write this simulation more concisely as 10+rnbinom(10000,10,0.02) – A. Webb Mar 7 '16 at 17:07
• @A.Webb thank you, that's a good point. I really was reinventing the wheel. I need to read ?rnbinom and then I'll edit my post – Adrian Mar 7 '16 at 20:38
• That would be 10/(10+rnbinom(10000,10,0.02)). The parameterization is in terms of number of successes/failures rather than total number of trials, so you'll have to add k=10 back. Note that the unbiased estimator would be 9/(9+rnbinom(10000,10,0.02)), one less in numerator and denominator. – A. Webb Mar 7 '16 at 20:47