# Are all balls in the urn the same color (when they can't be seen clearly)

I have a problem that reduces to balls in urns (it is actually about reference and alternate alleles in populations).

Assume I have a large well-mixed urn (i.i.d draws) that can contain two colors of balls: aquamarine and robin's egg blue (a and r respectively). They are close in color, so sometimes a person classifying them makes a mistake identifying the color after drawing a ball from an urn. Let $e_r$ be the probability of an error when the ball is really r and $e_a$ when the ball is really a. Assume I know these numbers (I think they're less than 0.01 but still need to check) and I have chosen a significance.

In an experiment, my companion draws $n$ balls from the urn and identifies $r$ balls as color r and $a$ as a ($n=r+a$). He then tells me $r$ and $a$. I want to test $H_0$ that all balls are r versus $H_a$ the urn contains at least one a ball given the numbers of balls drawn.

My goal is to perform the test at 2 different levels to give a "star" rating to the strength of the reported results. Could not reject at 0.05 = 2 stars, rejected at 0.05 = 3 stars and rejected at 0.01 = 4 stars.

What test can I use for this problem? (Though I've put this in conventional terms, I'd be happy with getting a Bayes factor and setting up thresholds based on that. I'm also happy with tests that require a certain number of measurements for validity - I can just classify samples that are too small as "could not reject")

Note this is different than testing a proportion because those tests don't have error in measurement (and don't work for proportion = 0 or 1). I thought of trying to set a non-zero $H_0$ proportion using some kind of fudge factor based on the error rate and the sample size(e.g. testing $H_0=P \le e_r$ where $P$ is the true proportion, but I couldn't come up with a well-justified number). I also started trying to derive my own test, but it was taking quite a while and this seems like the kind of problem someone would have investigated before.

Edit Rewrote the question slightly to clarify that I don't know the sequence of draws/classifications

I admit I didn't fully read the other answer, but a crude approach would just be to note that $a$ follows a binomial$(n, p = e_r)$ distribution when all the balls are robin's egg blue, so you can reject when $a$ is "too large" based on the binomial model. If this won't do then maybe a likelihood ratio test would be better, which seems to be what Zachary Blumenfeld is getting at.

I think I have the likelihood function (Full discloser, I am not 100% sure). Once you derive a likelihood though the rest of the hypothesis test should be easier.

Suppose you drew a sample of size $n$ denoted as $(X_1,...X_n)$. For simplicity, lets say;

$$X_i = \begin{cases} 1\;\;\mathrm{if \; classified \; as\; color}\;\;\mathbf{a}\\ 0\;\;\mathrm{if \; classified \; as\; color}\;\;\mathbf{r} \end{cases}$$ Further denote the "true" color indicator of observation $i$ as $X_i^*$ such that; $$X^*_i = \begin{cases} 1\;\;\mathrm{if \; observation\; \; is\; color}\;\;\mathbf{a}\\ 0\;\;\mathrm{if \; observation \; is\; color}\;\;\mathbf{r} \end{cases}$$ Suppose also that the error rate is known, $e_r \in (0,1)$.

The probability of $X_i$ conditional on $X^*_i$, is then a Bernoulli distribution; $$P(X_i=1|X^*_i,e_r)=\begin{cases} 1-e_r\;\;\mathrm{if}\;\;X^*_i=1\\ e_r\;\;\mathrm{if}\;\;X^*_i=0 \end{cases}$$ We can also express this as; $$P(X_i|X^*_i,e_r)=X^*_i\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-X^*_i)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg]$$ We also know the probability of $X^*_i$ $$P(X^*_i|p) = p^{X^*_i}(1-p)^{1-X^*_i}$$ and that $$P(X_i|e_r,p)=P(X_i|X^*_i=1,e_r)P(X^*_i=1|p) + P(X_i|X^*_i=0,e_r)P(X^*_i=0|p)$$ and then after some algebra; $$P(X_i|e_r,p) = p\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-p)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg]$$

So your likelihood is; $$\mathcal{L}(p\mid X_1,..,X_n,e_r)=\prod_{i=1}^n P(X_i|e_r,p)$$ $$= \prod_{i=1}^n p\bigg[(1-e_r)^{X_i} e_r^{1-X_i}\bigg]+(1-p)\bigg[e_r^{X_i} (1-e_r)^{1-X_i}\bigg]$$

Your hypothesis test reduces to $H_0: p=1$ vs $H_1: p\neq 0$. You can do that with a Bayes factor, or with standard error derived from the likelihood, or even via a parametric bootstrap. However you wish. Now that you have the likelihood, the rest should be easy.

• I see $P(X_i=1|X_i^*,e_r)$ but not $P(X_i=0|X_i^*,e_a)$. To do $P(X_i|...)$, I think you need both $e_r$ and $e_a$ – Eponymous Feb 25 '16 at 14:15
• Sorry, I assumed $e_r=e_a$ in this problem, so that will have to be changed. I also wrote the hypothesis the opposite way around (the null should be $p=0$, but that is easily changed as well. – Zachary Blumenfeld Feb 25 '16 at 18:31