Test whether probabilities of heads are under-estimated across many coins

Question

I am trying to figure out a problem that is equivalent to the following. Suppose you have a bag of $n$ coins in which each coin is labelled with a probability $p_i$ that it will come up heads when you toss it. You are given the result of $m$ coin-flips as well as which coin was flipped (hence the relevant $p_i$s each time. What I am do is test the hypothesis that the labels on the coins underestimate the probability of heads coming up.

At first I wanted to take a very simple approach to this and just simulate things but the results seem strange to me and I think something must be wrong. What I did was simulate $N$ sets of size $m$ draws using a fixed set of probabilities for each coin. I then calculated the likelihood of each of these draws using the probabilities given on the coins. Then, I drew a set 'test set' of coins from a with a different set of probabilities (in practice I set the probabilities of the 'test-set' of coins to be the values of the original set +0.2), calculated the likelihood and compared this to the 10000 simulations. When I do this I often find that the likelihood of the test sample is not different to that of the true data. While this is obviously totally possible, when I just run a fischer exact test on the failures and successes for the same simulation (or even just a binomial hypothesis test), the data comes out as different. Is this possible/likely? Am I doing something wrong? Is there an alternative way to do this hypothesis test?

Answer 1

If you toss $m$ coins then the number of heads follows a $X_i\sim B(1, \pi_i)$ distribution. Therefore the test statistics $X=\sum_{i=1}^m X_i$ follows a Poisson binomial distribution.

If $p_0=\sum_{i=1}^m p_i$ and $\pi$ is the true sum of probabilities then you could test $H_0: \pi\leq p_0$ vs. $H_1: \pi>p_0$ (underestimation case). There are efficient methods to compute $Pr(X=k)$ such that you can do even an exact test.