Skip to main content
2 of 4
added 162 characters in body
Carl
  • 13.2k
  • 7
  • 55
  • 115

The dirty coins and the three judges

All things being perfect, if we want to know if a set of coinage (all coins with the same bias) is biased, and to what degree, we would just use the outcomes and confidence intervals of the binomial distribution and be done with it.

Unfortunately, things are rarely that simple in real life. Indeed, this question arose as one of my concerns for what seems like a completely different problem elsewhere. If we are not sure what the true outcome is, and we have to adjudicate the results, that muddies the waters, and increases the confidence interval width. Now suppose we have dirty coins and we do not know for sure what is a heads or tails, or, some other similar problem, like having to use adjudication with three judges to say which image reconstruction method is more diagnostic (for example), then how well the judges agree with each other becomes a factor in whether we can detect that one method of image reconstruction is better than another or whether we can detect biased coinage.

Question: How do we determine, without having truth data, how sloppy our confidence interval is if we have three judges who do not always agree with each other as to what the outcome is?

Now, I do not know how to calculate this, but I can simulate it, and even the simulation has a curve ball. To that in a minute, first though, keep in mind that we do not have access to the hidden truth in the real world, all we know is probable truth. However, we can assign a synthetic truth when we simulate results.

The simulation

I did 10,000 binary random trials with 3 different biases of coinage, 0.5, (0.2, 0.8), and (0.1 0.9). Note that a bias of p=0.2 is symmetric with p=0.8, so that the 3 trials covers 5 cases in total. Next, as I had that as truth data, I could arbitrarily assign known reliabilities for our judges, and for each judge I simulated their decisions using two factors. For example, for a reliability (or correct knowledge, if you will) of 50%, I allowed the truth data to influence that judges decision 50% of the time. The next factor is that I randomized the remaining decisions, i.e., 50% in our example to be a yes 50% of the time (the latter is perfect cluelessness). That means that for a reliability of 50%, the judge will make a correct decision a factor of 0.5 + 0.5 p of the time. However, to increase the overall reliability of adjudication, we need more than one judge.

Now the curve ball. With three judges, I was able to contrast two strategies to adjudicate their decisions. These are 1) The median result, which is appealing because it is always a 0,1; yes or no, or 2) the mean result. Now let is look at the result for 10,000 trials of three adjudications each. enter image description here

In the figure, the $y$-axis represents R$^2$, i.e., explained fraction, and the $x$-axis represents the judges reliability. Note that the most accurate (i.e., efficient) results were had when coinage was 50-50 (p=0.5). However, that is also the least useful result in the sense of being able to detect the one method is better than the other for image reconstruction or for detecting coinage bias. When the three judges' reliability was >0.63, the median method outperformed the mean as an adjudication strategy. This became pronounced when the judges were very reliable (0.95). On the other hand, the mean became the better adjudication strategy for more unreliable judging (<0.63). Now results for single judges are not shown, they were miserable.

In the real world case, the judges might have different abilities, one might be 50% reliable and another 80%, depending on how 'expert' they are. Notwithstanding, for the real case, we only need some indication as to how to calculate our confidence intervals more or less accurately, and any insight into this interesting problem would be welcome.

Carl
  • 13.2k
  • 7
  • 55
  • 115