Setting null hypothesis for Binomial test

Question

I have a question for setting null hypothesis in binomial test. More specifically, is there any way to assume chance level when it is unknown?

For example, let's say that I have a slightly bent coin. So chance for getting Heads or tails is not equal. Instead, it's more likely that it will give us tail. To find out the probability, I tossed the coin 100 times, and got P(getting tail) = 68%

Then I bent it even more, and I wanted to check if it had any effect on the probability. So I threw it 10 times and got tails 9 times (90%).

Is it correct to use one sample binomial test to check if 9 out of 10 (90%) is significantly different using 68% as expected probability?

I'm not sure if it's okay because it "assumes" that chance level is 68% based on experience of throwing the coin 100 times, instead of mathematical calculation.

Thanks for reading.

Answer 1

No, it is not correct. In this example you have two samples so you should be using a two-sample hypothesis test. There are a few different two-sample binomial tests available, so you will need to choose one.

Answer 2

Is it correct to use one sample binomial test to check if 9 out of 10 (90%) is significantly different using 68% as expected probability?

I don't think this is correct because you would assume that 68% was measured without uncertainty. Intuitively, 68% obtained from 10000 flips is "better", more certain than 68% from just 100 flips.

In your case you could use Fisher test for the null hypothesis that 68/100 and 9/10 can originate from the same coin. In R:

mat <- matrix(c(68, 9, 100-68, 10-9), nrow= 2)
mat
     [,1] [,2]
[1,]   68   32
[2,]    9    1

fisher.test(mat)

    Fisher's Exact Test for Count Data

data:  mat
p-value = 0.2765
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.005230503 1.851113305
sample estimates:
odds ratio 
 0.2384069 
```

Answer 3

No, you treat the result from your sample of 100 ($\hat{p}=68/100$) as a sample statistic not a population proportion. If you repeated that trial of $100$ before the second bending, you would be very likely not to get $68$ on the second attempt; you'd see $65$ or $74$ or $70$ or $63$, or some other number, and a third trial would be likely different again from the first two.

Which is to say $0.68$ is just a noisy estimate of the true proportion. You should not ignore that uncertainty, treating it as if it were the true proportion.