I have a problem that reduces to balls in urns (it is actually about reference and alternate alleles in populations).

Assume I have a large well-mixed urn (i.i.d draws) that can contain two colors of balls: aquamarine and robin's egg blue (a and r respectively). They are close in color, so sometimes a person classifying them makes a mistake identifying the color after drawing a ball from an urn. Let $e_r$ be the probability of an error when the ball is really r and $e_a$ when the ball is really a. Assume I know these numbers (I think they're less than 0.01 but still need to check) and I have chosen a significance.

In an experiment, my companion draws $n$ balls from the urn and identifies $r$ balls as color r and $a$ as a ($n=r+a$). He then tells me $r$ and $a$. I want to test $H_0$ that all balls are r versus $H_a$ the urn contains at least one a ball given the numbers of balls drawn.

My goal is to perform the test at 2 different levels to give a "star" rating to the strength of the reported results. Could not reject at 0.05 = 2 stars, rejected at 0.05 = 3 stars and rejected at 0.01 = 4 stars.

What test can I use for this problem? (Though I've put this in conventional terms, I'd be happy with getting a Bayes factor and setting up thresholds based on that. I'm also happy with tests that require a certain number of measurements for validity - I can just classify samples that are too small as "could not reject")

Note this is different than testing a proportion because those tests don't have error in measurement (and don't work for proportion = 0 or 1). I thought of trying to set a non-zero $H_0$ proportion using some kind of fudge factor based on the error rate and the sample size(e.g. testing $H_0=P \le e_r$ where $P$ is the true proportion, but I couldn't come up with a well-justified number). I also started trying to derive my own test, but it was taking quite a while and this seems like the kind of problem someone would have investigated before.

Edit Rewrote the question slightly to clarify that I don't know the sequence of draws/classifications


2 Answers 2


I admit I didn't fully read the other answer, but a crude approach would just be to note that $a$ follows a binomial$(n, p = e_r)$ distribution when all the balls are robin's egg blue, so you can reject when $a$ is "too large" based on the binomial model. If this won't do then maybe a likelihood ratio test would be better, which seems to be what Zachary Blumenfeld is getting at.


I think I have the likelihood function (Full discloser, I am not 100% sure). Once you derive a likelihood though the rest of the hypothesis test should be easier.

Suppose you drew a sample of size $n$ denoted as $(X_1,...X_n)$. For simplicity, lets say;

$$ X_i = \begin{cases} 1\;\;\mathrm{if \; classified \; as\; color}\;\;\mathbf{a}\\ 0\;\;\mathrm{if \; classified \; as\; color}\;\;\mathbf{r} \end{cases} $$ Further denote the "true" color indicator of observation $i$ as $X_i^*$ such that; $$ X^*_i = \begin{cases} 1\;\;\mathrm{if \; observation\; \; is\; color}\;\;\mathbf{a}\\ 0\;\;\mathrm{if \; observation \; is\; color}\;\;\mathbf{r} \end{cases} $$ Suppose also that the error rate is known, $e_r \in (0,1)$.

The probability of $X_i$ conditional on $X^*_i$, is then a Bernoulli distribution; $$ P(X_i=1|X^*_i,e_r)=\begin{cases} 1-e_r\;\;\mathrm{if}\;\;X^*_i=1\\ e_r\;\;\mathrm{if}\;\;X^*_i=0 \end{cases} $$ We can also express this as; $$ P(X_i|X^*_i,e_r)=X^*_i\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-X^*_i)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg] $$ We also know the probability of $X^*_i$ $$ P(X^*_i|p) = p^{X^*_i}(1-p)^{1-X^*_i} $$ and that $$ P(X_i|e_r,p)=P(X_i|X^*_i=1,e_r)P(X^*_i=1|p) + P(X_i|X^*_i=0,e_r)P(X^*_i=0|p) $$ and then after some algebra; $$ P(X_i|e_r,p) = p\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-p)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg] $$

So your likelihood is; $$ \mathcal{L}(p\mid X_1,..,X_n,e_r)=\prod_{i=1}^n P(X_i|e_r,p) $$ $$ = \prod_{i=1}^n p\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-p)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg] $$

Your hypothesis test reduces to $H_0: p=1$ vs $H_1: p\neq 0$. You can do that with a Bayes factor, or with standard error derived from the likelihood, or even via a parametric bootstrap. However you wish. Now that you have the likelihood, the rest should be easy.

  • $\begingroup$ I see $P(X_i=1|X_i^*,e_r)$ but not $P(X_i=0|X_i^*,e_a)$. To do $P(X_i|...)$, I think you need both $e_r$ and $e_a$ $\endgroup$
    – Eponymous
    Feb 25, 2016 at 14:15
  • $\begingroup$ Sorry, I assumed $e_r=e_a $ in this problem, so that will have to be changed. I also wrote the hypothesis the opposite way around (the null should be $p=0$, but that is easily changed as well. $\endgroup$ Feb 25, 2016 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.