I have a question related to hypothesis testing. I cannot seem to find error in my reasoning.

Initial question: We have 230 patients: 157 males and 73 females. 142 males are sick and 69 females are sick. Can we say that females are sick "statistically significantly" more?

My method:
Assume that females and males come from binomial distributions. Females are sick with probablity $p_\text{female}$, males - $p_\text{male}$.

We want to test that $p_\text{female} = p_\text{male} = p_\text{overall}$

Given that null hypothesis is true, the probability of observing such "extreme" results (so many females and so little males being sick) is:

cdfbinom(69, 73, prob=211/230)) * (1-cdfbinom(142, 157, prob=211/230)

Which is equal to 0.04351274, therefore we reject the hypothesis that they are the same.

I cannot understand where is my mistake, as all of the other methods: chi-square test, t-test, even running logistic regression where I include gender indicator as a variable, do not reject the null hypothesis.


3 Answers 3


You are only considering one-sided extremities.

In your probability statements, you should also include the 'extreme cases' of having 'few women' and 'many men'. This will surely raise the probability total, and will likely be above your default threshold (0.05) since it's already close.

  • $\begingroup$ i would use a chisq test to test independence of the factors sex and sick.. Is this a right approach? $\endgroup$
    – user974514
    Mar 25, 2013 at 17:06
  • $\begingroup$ Thanks for your answer, Nick. However, the problem is that the question is asking if females are sick "more often", not "if the sickness rates differ" - is my method of testing correct then? Also, another confusing part is that in all the other tests I get p-values around 0.3, and in this case even if I would include "both extremities" the discrepancy between other tests is still rather big... $\endgroup$
    – Vainius
    Mar 26, 2013 at 13:01
  • 1
    $\begingroup$ OK, I looked into this a little deeper (but only a little). I was misled on the one-sided part because you compare your result to a chi-square test, which is, to my knowledge, always two-sided (and thus will always have a bigger p-value). I've no clue which t-test you tried to perform, so I'll stay away from that. Default outputs from most logistic regressions will also show two-sided p-values. $\endgroup$
    – Nick Sabbe
    Mar 26, 2013 at 14:47
  • 1
    $\begingroup$ As for your own suggested p-value: at the very least, it's not so obvious that you can just plug in your estimate (211/230) for the population proportion in the cumulative probabilities. $\endgroup$
    – Nick Sabbe
    Mar 26, 2013 at 14:50
  • $\begingroup$ I have also tried pluging 'optimal' value (in sense that it gives maximal p_value) for it and it was very close to 211/230. Yes, but it is still not obvious if I can call the number I get a "p-value"... $\endgroup$
    – Vainius
    Mar 26, 2013 at 15:43

There are several problems with this approach, here are two:

  • Each part of your computation is basically the p-value from a one-tailed one-sample test that the population proportion is different from 211/230. This does not directly test for the difference between groups and also ignores the uncertainty in this number (i.e. 211/230 is not some theoretical value or a priori hypothesis, it's something you computed on the same data).
  • You cannot combine $p$-values in this way. One intuition to show this: The cumulative distribution function and $p$-values are bounded between 0 and 1; if you multiply several $p$-values together you will inevitably end up with increasingly smaller numbers and eventually cross whatever threshold you set. Consequently, you could take any sample (including pure noise or random data in which the null hypothesis is true by construction), slice it in small batches, run a test on each batch and multiply the $p$-values together to obtain a very small number. What could it possibly mean?

The way you describe the problem, you can consider a contingency table:

Health Male Female
Sick 142 69
Healthy 15 4

Let us call $\theta$ the underlying probability of being sick (mainly to avoid so many $p$s). Several exact tests use the binomial distribution to check he null hypotheses $\theta_{female} = \theta_{male}$ or $\theta_{male} > \theta_{female}$. You can read about Fisher's exact test or Barnard's test. You can also consider using some variation of the $\chi^2$ test if the number of observations is large.

In addition, I have recently described an exact test to check any of these hypotheses in R (the package is here). Let us set $\alpha$ to 0.05 and test both hypotheses:

$\theta_{female} = \theta_{male}$

m.test(list(c(69, 4), c(142, 15)))
[1] 0.7113878

The null hypothesis is not rejected.

$\theta_{male} > \theta_{female}$

os.m.test(list(c(142, 15), c(69, 4)))
[1] 0.0101545

And the null hypothesis is rejected. In this case, this does not necessarily mean that $\theta_{female} > \theta_{male}$. As seen in the previous operation, the results are compatible with $\theta_{female} = \theta_{male}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.