Hypothesis testing: Fisher's exact test and Binomial test

Question

My first post in this community. My knowledge of statistics is limited, and so I am seeking advice on the following problem. (Sorry for the long post, and thank you in advance for your help).

I have a group of 15 children with taste disorders (a not so common condition); 8 are girls, and 7 are boys.

I do the following experiment. Each child in my group tastes a slice of a specially prepared cake, and then I ask the child "does the cake taste good?", and the child can answer either "yes" or "no". Each child tastes the cake separately, and answer my question before meeting any other child in the group.

I collect the following data:

	cake tastes good	cake does not taste good
girls	7	1	8
boys	6	1	7
	13	2	15

1. Considering the population of girls with tastes disorders, I do a binomial test with number of success k = 7, number of trials n = 8, and probability of success p = 0.5, to test my null hypothesis H0 = "my cake tastes good for no more than 50% of the population of girls with taste disorders". In python I can run binomtest(7, 8, 0.5, alternative="greater") which gives the following result BinomTestResult(k=7, n=8, alternative='greater', proportion_estimate=0.875, pvalue=0.03515625) and ConfidenceInterval(low=0.5293205913988617, high=1.0). I find that p-value <= 0.05, and therefore I can reject H0, and say that "my cake tastes good for more than 50% of the population of girls with taste disorders".

2. Similarly, considering the population of boys with tastes disorders, I can do a binomial test to test my null hypothesis "my cake tastes good for no more than 50% of the population of boys with taste disorders". In python I can run binomtest(6, 7, 0.5, alternative="greater") which gives the following result BinomTestResult(k=6, n=7, alternative='greater', proportion_estimate=0.8571428571428571, pvalue=0.0625). I find that p-value > 0.05, and therefore I cannot reject H0, and I say that "my cake tastes good for no more than 50% of the population of boys with taste disorders".

3. Now I run a Fisher's exact test on my contingency table. My null hypothesis is H0 = "there is no significant difference between the proportion of girls with taste disorder who find that my cake tastes good, and the proportion of boys with taste disorders who find that my cake tastes good". In python I can run fisher_exact([[7, 1], [6, 1]], alternative="two-sided") which gives the following result (1.1666666666666667, 1.0), where the fist value (1.17) is the odds ratio, and the second value (1) is the p-value. I find that p-value >= 0.05, and therefore I cannot reject the null hypothesis, and I say that "there is no significant difference between the proportion of girls with taste disorder who find that my cake tastes good, and the proportion of boys with taste disorders who find that my cake tastes good".

The result obtained with the Fisher's exact test ("no significant difference between the proportion of girls and boys who finds that the cake tastes good") seems to contradict the results in (1) and (2), which say that the "more than 50% of the population of girls find that the cake tastes good" (1), and "no more than 50% of boys find that the cake tastes good" (2). How do interpret these results?

[problem re-phrased (hopefully in a better way) according to whuber suggestion]

Answer 1

I believe you are misinterpreting the results. Statistical tests in general don't give you a yes/no answer, but only likely/not so likely (given the data).

The non-significant result in your experiment (2) (boys tasting cake), $p = 0.0625$, does not mean:

We know for sure that no more than 50% of the boys in the population would find that the cake tastes good.

Instead, you'd better interpret it as:

Based on the available data, we cannot conclude, with the desired certainty, that the cake would taste good to more than 50% of the boys in the population.

It still might (and likely does), but you lack the evidence. If you had more data, you could come to that conclusion, even if the ratio remained the same. For example, imagine having a twice as big sample, 14 boys, of which 12 find the cake tasty. The ratio, $12 / 14 = 6 / 7$, is the same, but the binomial test would give you $p \approx 0.0065$, i.e. significant.

The $H_0$ you work with in the binomial test is that $P($tasty$) = 0.5$. In Fisher's exact test, you have a different hypothesis. You assume that the ratio good/bad is $13/15$, regardless of the sex, and ask whether the observed ratio for the boys, $6/7$ significantly differs from that. It doesn't, but, again, if you had more data, it might:

fisher_exact([[6000, 1000], [7000, 1000]])

results in $p \approx 0.0014$.

Answer 2

You are testing two different hypotheses

• With the binomial test the hypothesis is that the proportion is 0.5
• With the Fisher exact test the hypothesis is that the proportion is the same for boys as for girls.

which say that the "more than 50% of the population of girls find that the cake tastes good" (1), and "no more than 50% of boys find that the cake tastes good

That is not what the test says.

The tests says that among the girls 87.5% finds that the cake tastes good, and among the boys 85.7% finds that the cake tastes good. The difference between the two is that the effect found in the girls group is statistically significant.

Answer 3

As was already stated in other answers, tests are not interpreted in the way you do in the question.

A test addresses the question whether there is clear evidence that a certain model is not appropriate. If such evidence cannot be found, it doesn't mean that the model is correct, it only means that the data do not indicate clearly that it is wrong.

Let's look at the data. I'll first discuss Fisher's test. The model of interest here is a model in which the probability for girls and the probability for boys to like the cake is the same. For girls, 7 out of 8 like it, for boys, 6 out of 7 like it. For sure, if you think about it, if both have the same probability to like the cake (0.9 for example, but it may well be 0.8 or 0.7), such a result can be realistic, there's nothing suspicious about it. Consequently, Fisher's test does not reject the model. This does not mean that the model is true. The data may well also occur if the probability for girls is 7/8 (or maybe 90%) and that for boys is different, for example 6/7 (or maybe 70%). Under all these choices, the data do not look unusual at all, so none of these would be rejected.

The thing to understand here is that there are always many models and choices of parameters compatible with the data, and hence cannot be rejected.

Let's now look at the binomial test for the 50% hypothesis for the girls. In fact, 7 girls liked the cake and 1 didn't. That's a pretty extreme result for a 50% probability. Is it actually compatible with a true population probability of 50%? Well, the test gives p=0.035, meaning that such an extreme result (or even more extreme) is only expected to occur, under the 50%-model, in 3.5% of cases. That's pretty rare, and surely it raises suspicion against the model. The suspicion is not extremely strong, things with probability 3.5% happen from time to time, but it is very legitimate now to doubt the model. (If you "reject the null hypothesis at level 5%", it basically means that you are not willing to entertain as a realistic option a model under which what just has happened would happen only in 1/20 of experiments or even less often.)

For boys, 6 liked the cake and 1 didn't. Still a rather extreme result, so you might still be skeptical about the 50% null hypothesis. The p-value tells you that something like this or even more extreme will happen in 6.25% of cases. Still a quite low number, and doubts about the model are perfectly justified. But it's almost twice as the corresponding probability for the girls, so although the result looks somewhat suspicious and would in fact still be rare under the model, it is clearly less striking than for the girls, and a statistician committed to a 5% significance level might say, the result is somewhat surprising but it is still possible enough that I am not fully convinced the model is wrong, despite of course knowing that it is very possible that it is wrong (probabilities of 70, 80, or even 90% are as well compatible with the data, see above).

So these results and p-values can be connected to some straightforward direct intuition about the data: "imagining that girls and boys probability are the same, this looks quite normal and realistic; however thinking about a 50% probability for girls, the result looks very suspicious and unrealistic; thinking about a 50% probability for boys, the result looks somewhat suspicious, but not as much as for the girls."

Particularly, just that suspicion is very strong for girls and less strong for boys (owing to the smaller sample size by the way) doesn't imply at all that probabilities for girls and boys must be different! It means that for girls we can exclude 50% as an option at 5%-level, for boys we can't, but still, this doesn't affect at all the possibility that in fact it's something like 80% and equal for both of them... or slightly different... never forget, many models can be compatible with the same data!

Answer 4

Narrowing the question on my own, I would be interested whether, among children with taste disorder, liking the taste of your cake is independent of gender. [For example, if the disorder were color-blindness, it might make sense to look for gender differences.]

In R, Fisher's Exact test looks like this:

TBL = rbind(c(7,1), c(6,1))
fisher.test(TBL)

        Fisher's Exact Test for Count Data

data:  TBL
p-value = 1
alternative hypothesis: 
 true odds ratio is not equal to 1
95 percent confidence interval:
   0.01276832 104.39935860
sample estimates:
odds ratio 
  1.154699 

Here, Fisher's Exact test would be a traditional alternative to a chi-squared test of independence on a a $2 \times 2$ table. The table has too few subjects disliking the cake to give a reliable P-value using a chi-squared test:

chisq.test(TBL)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TBL
X-squared = 9.9048e-32, df = 1, p-value = 1

Warning message:
In chisq.test(TBL) : 
 Chi-squared approximation may be incorrect

Specifically, the warning message appears when any of the expected counts in the chi-squared test is below 5. Here are expected counts corresponding to your observed counts.

chisq.test(TBL)$exp
         [,1]      [,2]
[1,] 6.933333 1.0666667
[2,] 6.066667 0.9333333
Warning message:
In chisq.test(TBL) : 
 Chi-squared approximation may be incorrect

As implemented in R, one can sometimes get a more useful P-value for the chi-squared test by simulation; simulation makes no difference in the reported P-value for your data.

chisq.test(TBL, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 0.010302, df = NA, p-value = 1

Answer 5

The p-value is the probability that the result you observed would have been produced under the null hypothesis, i.e. if in fact 50 percent or less girls think the cakes tastes good (from some theoretical population), then if you randomly sampled that population and had eight girls take the test, the probability that seven would like it. This is the same probability of flipping a fair coin eight times and getting seven heads. It's pretty unlikely.

The probability of six heads out of seven flips is slightly higher, but just because this probability is above 0.05 and the other is below 0.05 doesn't necessarily mean much, especially when you're dealing with such a small sample size.

There's nothing special about 0.05. The difference between statistically significant results and non-statistically significant results is not necessarily statistically significant itself, i.e. may we just be an artifact of chance.