I would appreciate so much if someone could help me to understand the following comparisons, and tell me if I am doing the tests correctly. I want to compare two districts and their respective proportions of white over red houses. Why the pvalues are different? (please READ the EDIT below)

Dataset 1 (difference in 10% between districts):

house <- matrix(c(50,50,60,40), ncol=2, dimnames=list(c("district_A", "district_B"), c("white","red")), byrow=TRUE)
prop.table(house, margin=1)

So here a sample size of 100 and a difference between districts in 10% leads to p=0.201

Dataset 2: (still same difference of 10% between districts, but now the respective proportions are different):

house2 <- matrix(c(60,40,70,30), ncol=2, dimnames=list(c("district_A", "district_B"), c("white","red")), byrow=TRUE)
prop.table(house2, margin=1)

Here a sample size of 100 and a difference between districts in 10% leads to p=0.182

Thank you very much for your kind help.


If I plot the pvalues obtained from the fisher test run on 10 datasets (10 white/90 red over 20white/80 red, 20white/80red over 30white/70 red, 30white/70red over 40white/60 red, etc.) having the same sample size (n=100) and within each dataset the difference between the two districts is 10%, this is what I obtain:

bell curve of pvalues

So it means that, for a fixed % difference between two groups, the closer one proportion is to 0.5, the higher is the p-value. I don't understand then, how we can rely on this as in a dataset we might conclude for a significant difference between proportions of 0.8 vs 0.9, whereas we might conclude for a NON-significant difference between proportions of 0.5 vs 0.6, assuming an equal sample size.


The ability to distinguish 2 proportions depends on the precision with which you can estimate the true proportions from the observed data. That's related to the variance.

Consider just a single population with 2 groups and a true proportion $p$ in group 1 and thus $(1-p)$ in group 2. If you take a sample of size $n$ from that population, the variance of the associated binomial distribution is:

$$ np(1-p) $$

So for a 50/50 ($p=0.5$) split between the groups, the variance is $.25 n$. For a 70/30 split, the variance is smaller: $.21n$. For a 90/10 split, it's $0.09n$. The highest variance is for the 50/50 split between the two groups.

So the ability to detect a certain difference in terms of percentage points gets easier as you move away from a 50/50 split, as you found.

| cite | improve this answer | |
  • $\begingroup$ Thank you very much! This explains a lot! But it also confirms me that we should be very careful by using these tests, this "kind-of-bias" sounds to me to constitute an important limitation of their use in many area $\endgroup$ – Peter May 26 at 12:14
  • $\begingroup$ @Peter it's not really a "bias" in the statistical sense. This is an inherent property of sampling from a binomial distribution. What can get confusing is that simple intuition based on what you learned about analysis of variance or ordinary least squares regression (under their usual assumptions) can lead you astray when analyzing binomially distributed data. $\endgroup$ – EdM May 26 at 15:53

You are doing the tests correctly, mechanically. But you may or may not be doing the test you want depending on your research question. And you are certainly wrestling with what a Fisher Test does.

May I suggest checking Stat's Exchange here for more info and perhaps a follow-on question there.

| cite | improve this answer | |
  • $\begingroup$ Ok thank you. So depending on the value of the proportion, with a same % of difference between the two groups, the test will have different pvalues. It sounds weird to me. $\endgroup$ – Peter May 18 at 15:43
  • $\begingroup$ My research question would be: Does the geography (ie, district A vs district B) has an impact on the proportion of houses being "white"? And houses can be only white or red (binomial data). $\endgroup$ – Peter May 18 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.