Calculating confidence interval for whether some of N coins are unfair

Question

I'm having trouble figuring out the appropriate way to place a confidence interval on a ratio I'm estimating. It's for a social science application, but I'll present the problem via coin-flips as I think its clearer.

Imagine you have $N=100$ coins, and you are going to flip each coin 3 times. Your null hypothesis H0 is that all the coins are fair, $p=q=.5$. But you suspect (H1) that actually some of the coins are fake such that they always return heads (which we will set equal to 1). You flip each coin three times, such that your results are N binomial variates of size=3.

I can observe the proportion of 'all heads' trials out of the N coins, and I can also calculate the predicted number of 'all heads' trials under the null hypothesis, which is $p^3N$. Say I observe 16 'all heads' trials. Under the null I expected to observe $(.5\times.5\times.5)N=12.5$ 'all heads' trials - how can I tell if these numbers are meaningfully different?

I know that the variance is $Npq$ for the proportion of successes $n$ in $N$ coin flips, but that doesn't seem the right way to go. In particular, the expected value of my result is $p^3N$, should I be treating an 'all heads' outcome like a success in a bernoulli trial of size $N$, and thus calculate variance as $Np^3(1-p^3)$?

Answer 1

Off the top of my head:

H0 : all coins are fair
H1 : at least one gives only heads

I see it as a binomial where the experiment is "throwing a coin 3 times", with N=100, just like you suggested.

Hence the number of "3 heads" $X$ follows a binomial distribution. $$ X \sim B(n=100,\; p=1/8) $$

This can be approximated by a normal law $$ N(\mu=pn,\; \sigma² = p(1-p)n) $$

So if you want say a 95% confidence interval (z=1.645), you will reject H0 if $$ X > \mu + z \sigma = 17.94 \approx 18$$

So if when you get 18 or more "3 heads" you conclude "something is fishy!!", you'll be wrong only about 5% of the time.

Answer 2

I am late to this thread, which already has a good answer, but an approximation (normal distribution) can leave you just as unsatisfied as the feeling of "cheating" when using the raw power of your computer to carry out a Monte Carlo simulation... So here is an alternative, less mathematical, but equally efficient and complementary, way of getting your $95\%$ confidence interval.

# Let's define a function to "toss" a coin x number of times:    
coin_tosses = function(x){sample(c(0,1), x, replace = T)}
# The next function reproduces tossing a coin x times (3) and repeating the  
# process n number of times (100) counting how many times we get 3 heads:
    experiment = function(x, n){
    n_times <- replicate(n, coin_tosses(x))
    results <- apply(n_times, 2, sum)
    mean(results==3)
    }
# Now we do this many times... 10,000 for example:
    results <- replicate(1e4, experiment(3, 100))
# Plot the results:
    hist(results, border = F, col="gray80", main = "Proportion 3 Heads")
# And calculate the 95% confidence interval:
    quantile(results, probs = c(0.025,0.975))

And the $\text{CI}_{95\%}= (0.25\% =0.06;\,\,97.5\%= 0.19) $.

Here is the plot...

And, just for fun, we can also calculate a $\text{p}$-value for any given proportion of $3\text{H}$ results we may get:

# This is the function    
p_value = function(x) mean(results > x)
# So, What is the p-value of getting a proportion of, for instance 20% of 3H results:
    p_value(0.20)    [1] 0.0102