Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Suppose we have a Bernoulli process with failure probability $q$ (which will be small, say, $q \leq 0.01$) from which we sample until we encounter $10$ failures. We thereby estimate the probability of failure as $\hat{q}:=10/N$ where $N$ is the number of samples.

Question: Is $\hat{q}$ a biased estimate of $q$? And, if so, is there a way to correct it?

I'm concerned that insisting the last sample is a fail biases the estimate.

share|improve this question
5  
The current answers stop short of providing the minimum variance unbiased estimator $(10-1)/(N-1)$. See sampling and point estimation section of the Wikipedia article on the negative binomial distribution. – A. Webb Mar 7 at 18:31
up vote 9 down vote accepted

It is true that $\hat{q}$ is a biased estimate of $q$ in the sense that $\text{E}(\hat{q}) \neq q$, but you shouldn't necessarily let this deter you. This exact scenario can be used as a criticism against the idea that we should always use unbiased estimators, because here the bias is more of an artifact of the particular experiment we happen to be doing. The data look exactly as they would if we had chosen the number of samples in advance, so why should our inferences change?

Interestingly, if you were to collect data in this way and then write down the likelihood function under both the binomial (fixed sample size) and negative binomial models, you would find that the two are proportional to one another. This means that $\hat{q}$ is just the ordinary maximum likelihood estimate under the negative binomial model, which of course is a perfectly reasonable estimate.

share|improve this answer
    
Great! It looks like (for my purposes) this bias is not an issue. – becky Mar 7 at 8:38

It is not insisting that the last sample is a fail which biases the estimate, it is taking the reciprocal of $N$

So $\mathbb{E}\left[\frac{N}{10}\right] =\frac{1}{q}$ in your example but $\mathbb{E}\left[\frac{10}{N}\right] \not = q$. This is close to comparing the arithmetic mean with the harmonic mean

The bad news is that the bias can increase as $q$ gets smaller, though not by much once $q$ is already small. The good news is that bias decreases as the required number of failures increases. It seems that if you require $f$ failures, then the bias is bounded above by a multiplicative factor of $\frac{f}{f-1}$ for small $q$; you do not want this approach when you stop after the first failure

Stopping after $10$ failures, with $q=0.01$ you will get $\mathbb{E}\left[\frac{N}{10}\right] = 100$ but $\mathbb{E}\left[\frac{10}{N}\right] \approx 0.011097$, while with $q=0.001$ you will get $\mathbb{E}\left[\frac{N}{10}\right] = 1000$ but $\mathbb{E}\left[\frac{10}{N}\right] \approx 0.001111$. A bias of roughly a $\frac{10}{9}$ multiplicative factor

share|improve this answer

As a complement to dsaxton's answer, here are some simulations in R showing the sampling distribution of $\hat{q}$ when $k=10$ and $q_0 = 0.02$:

n_replications <- 10000
k <- 10
failure_prob <- 0.02
n_trials <- k + rnbinom(n_replications, size=k, prob=failure_prob)
all(n_trials >= k)  # Sanity check, cannot have 10 failures in < 10 trials

estimated_failure_probability <- k / n_trials
histogram_breaks <- seq(0, max(estimated_failure_probability) + 0.001, 0.001)
## png("estimated_failure_probability.png")
hist(estimated_failure_probability, breaks=histogram_breaks)
abline(v=failure_prob, col="red", lty=2, lwd=2)  # True failure probability in red
## dev.off()

mean(estimated_failure_probability)  # Around 0.022
sd(estimated_failure_probability)
t.test(x=estimated_failure_probability, mu=failure_prob)  # Interval around [0.0220, 0.0223]

It looks like $\mathbb{E}\left[ \hat{q}\right] \approx 0.022$, which is a rather small bias relative to the variability in $\hat{q}$.

histogram of q_hat

share|improve this answer
1  
That's really helpful. At that level, it's not worth me worrying about. – becky Mar 7 at 8:40
2  
You can write this simulation more concisely as 10+rnbinom(10000,10,0.02) – A. Webb Mar 7 at 17:07
    
@A.Webb thank you, that's a good point. I really was reinventing the wheel. I need to read ?rnbinom and then I'll edit my post – Adrian Mar 7 at 20:38
1  
That would be 10/(10+rnbinom(10000,10,0.02)). The parameterization is in terms of number of successes/failures rather than total number of trials, so you'll have to add k=10 back. Note that the unbiased estimator would be 9/(9+rnbinom(10000,10,0.02)), one less in numerator and denominator. – A. Webb Mar 7 at 20:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.