Edit Note: While this question is very interesting and relevant in its own right, I have come to a realisation that I have to make it a bit more complicated in order for it to be applicable to my actual research question. In order to avoid confusion, I have asked it as a follow-up question.
There is a device that produces coins. Most of the time it makes unbiased coins, but sometimes it makes biased coins. Let $q$ be the probability of producing a biased coin. Biased coins can have different bias. There is no prior knowledge about the distribution of these biases. However, in practice we frequently see strongly-biased coins.
We use the device to produce $N=100$ coins. Then we toss each coin $M=200$ times and record the results. We would like to test if $q > 0$, with the null hypothesis that $q=0$. We would like to estimate $\hat q$ and get a confidence interval for it.
Clearly, the problem is unsolvable if we allow for coins of arbitrarily small but non-zero bias. In order to circumvent this problem, we will only be counting the strongly biased coins.
Given the bias of the coin $b$ we will define a coin as strongly-biased if $|b-0.5|\geq 0.05$. So the goal is to estimate the fraction of strongly biased coins $\hat q'$ that the device produces, as well as its confidence interval.
Attempt: I have attempted a non-parametric approach. It seems to work in practice, but I'm not sure if the mathematics behind it is solid, or if it is the most powerful test I can do.
- For each coin, perform a binomial test against the null hypothesis, obtain a p-value
- Threshold each of the obtained p-values to $T = 5\%$, obtain a list of 1s and 0s
- Perform a binomial test on that list, with null hypothesis $P=T$ and alternative hypothesis $P > T$.
The logic behind it is that by testing multiple coins, some of them will test positive by chance. In order to answer the first question, we test if the number of coins we have classified as biased significantly exceeds that chance. My worry about this solution is that the threshold is somewhat arbitrary, that I don't know how it relates to the above definition of strongly-biased coins, and that I don't have a confidence interval.
It would be cool to alternatively try a model-based approach, addressing the hierarchical nature of this random process directly. However, I know nothing at all about the parametric methods used to tackle such problems. Any suggestions, names, links to literature are appreciated.
Note: This is a minimal example of a problem I have encountered in experimental design in neuroscience. I have judged that further details are unnecessary to make progress on this problem.