Assume we have two coins replicates, which are produced in the same pipeline. To test whether the coins replicates are both biased in one direction (e.g. the probability of head > 0.5), we did flip experiments on the two replicates separately. My question is how got the p-value based on the observations. The following R code is the two approaches I've tried. But I'm not confident they are right.

exp1 = 9 #total number of flips for coin 1.
head1 = 7 #number of heads
exp2 = 8 
head2 = 6

First approach: sum the two experiments together. But it ignores the variance of two coins.

binom.test(head1 + head2, exp1 + exp2)
#p-value = 0.04904

Second approach: try to calculate the joint probability of two experiments.

#Generate the probability distribution of each experiment
dy =  dbinom(x = 0:exp1, exp1,prob = 0.5)
dy2 = dbinom(x = 0:exp2, exp2, prob = 0.5)

observed_prob =dy[head1 + 1] * dy2_pvalue[head2 + 1]

#Calculate the joint probability of all the events given the number of trials
joint_density = as.matrix(data.frame(dy)) %*% as.matrix(t(data.frame(dy2)) )

#p-value = probability sum of the events with lower probability than the observed
sum(joint_density[joint_density < observed_prob ])

I'm surprised that the p-value of second approach is so different from approach 1. Am I wrong?

Edit after post

The goal is to find the pipeline which produces biased coins. So I think my hypothesis is:

H0: p1=p2=0.5 (p1 is the probability of head in coin 1)

H1.1: p1<0.5,p2<0.5 or p1>0.5,p2>0.5

Because two replicates are supposed to be consistent, so I prefer H1 to be:

H1.2: p1=p2<0.5 or p1=p2>0.5.

What should I do in case of H1.2 considering the variance of two replicates?

Note The number of flips of each coin can be very different(e.g. 30 flips for coin 1 and 3 flips for coin 2). So approach 1 would be heavily dominated by coin 1. This is why I want to try approach 2.

  • $\begingroup$ (1) By what mechanism could the two experiments possibly not be independent? (2) The replicates are both biased in one direction if and only if each is biased in that direction. Why not, then, conduct separate tests on each coin? $\endgroup$ – whuber Jul 17 '16 at 14:12
  • $\begingroup$ @whuber (1) Two experiments should be independent. (2) separate tests would give me two p-values. The question is how to get a single p-value from two? $\endgroup$ – user3915365 Jul 17 '16 at 15:02
  • $\begingroup$ Search our site for keywords like "multiple comparisons" and "Bonferroni." $\endgroup$ – whuber Jul 17 '16 at 15:10
  • $\begingroup$ @whuber what if one test passed the FDR threshold and the other not? $\endgroup$ – user3915365 Jul 17 '16 at 15:16
  • $\begingroup$ Here's it's important that they're coming from the same pipeline. You're testing whether the pipeline produces biased coins. Hence, the experiment on two coins is not independent in some sense. For instance, when I'm looking at the 1st coin it's reasonable to start with null that it's unbiased. However, if the experiment comes negative, then before starting trials with 2nd coin is it reasonable to start with unbiased assumption? $\endgroup$ – Aksakal Jul 17 '16 at 16:51

From your wording, I'm not very sure about exactly what are you trying to test.

If both series come from the coin, then we can take for granted that the probability of heads is the same for both and you can test if that probability is 0.5 (e.g. if the coin is biased). That is:

$$ H_0: p=0.5 $$ $$ H_1: p<>0.5 $$

That is the test you are performing with your code:

binom.test(head1 + head2, exp1 + exp2)
#p-value = 0.04904

If you are just interested in whether the coin is biased towards heads, then the alternative hypothesis would be $p>0.5$ and in your code you would use alternative="greater".

Please notice that here taking in account that we have two replicates is quite meaningless, since we assume that we are tossing the same coin and probabilities remain constant. Therefore, using both samples as a bigger single sample, as you did, is right.

Of course, if we suspect that something has changed between the two replicates (e.g. we used different coins, or the coin might have been altered), we could test if the probability in both replicates is the same. Then the test would be:

$$ H_0: p_1=p_2 $$ $$ H_1: p_1<>p_2 $$

However, this doesn't seem to be what you are searching for.

About your second approach you seem to be making a different and a bit strange test. Your hypothesis are something like:

$$ H_0: p_1=p_2=0.5 $$ $$ H_1: p_1<>0.5\text{ or }p_2<>0.5\text{ or }p_1<>p_2 $$

And for p-value you are summing probabilities of all combinations of head1 and head2 less probable than the one you get in your sample, which looks to me a quite strange test statistic, although for one sample it can be equivalent to the a two sided bilat.test. Furthermore, there are extra reasons to get very different p-values in your two approaches - or maybe just extra insights on it:

  • In the second approach you are dropping the assumption that both probabilities are equal. When you drop any assumption, the power of the test is reduced and usually the p-value is increased. With such an small sample, the p-value can be dramatically increased.
  • You can check which pairs (head1,head2) are you counting in each approach, and you will see a lot of pairs included in your second approach that are left in the first one. For example, a pairs like (head1=9,head2=0) or (head1=2,head2=7) are very likely to be included to compute p-value in your second approach but not for the first.

In summary, you get different p-values in your two approaches because you are performing different tests: the first one that fits what I suppose you are looking for, and the second one that is quite strange.

  • $\begingroup$ I added the hypothesis in the question. If I prefer the alternative hypothesis to be: p1=p2<0.5 or p1=p2>0.5. How should I do ? $\endgroup$ – user3915365 Jul 17 '16 at 15:40
  • $\begingroup$ In that alternative hypothesis you state that p1=p2, and you state the same in the null hypothesis. If that is what you want to mean, the only thing to be tested is the value of the common p. That is just the well known bilat.test you performed in your first approach. H0: p=0.5; H1: p<>0.5 However, if you didn't want to assume that both probabilities are equal, then you could want to test if they are biased in the same direction. Here I would prefer an alternative hypothesis like H1: (p1-0.5)*(p2-0.5)>0, and that is not standard (as far as I know) but we could figure how to perform it. $\endgroup$ – Pere Jul 18 '16 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.