Hi all and thanks for taking the time to read this question. I'll try to be as clear as possible!

The data: I have one sample of patients which was divided in two groups based on a binary variable for a condition - Condition A and Condition B. Gorup A contains approx. 500 patients, while B contains approx. 170. The patients records have a categorical variable with 5 different (mutually exclusive) outcomes, say $[o_1,o_2,o_3,o_4,o_5]$.

The problem: I want to compare the proportions of each of those outcomes between the groups, i.e., I want to know if the proportion of outcome o1 is different between groups A and B, AND I want to know if the proportion of outcome $o_2$ is different between A and B, and so on. There would be multiple $H_0$'s in this case, in the line of: "Outcome $o_1$ is the same between groups".

Additional details: Some of the outcomes have very small observed proportions (e.g. 1 in 170, 7 in 500).

My thoughts: I've compared groups like this before, but never had to determine which outcome was different, only if the overall outcome proportions were dependent on group (Fisher's test and Chi-Squared test, etc). The groups do form contingency tables, since a patient cannot have more than one outcome. Is there any specific test for this situation? Or should I use something like a Chi-Squared test for each outcome and then perform a multiplicity adjustment? I found this thread, which seems to be similar, however I couldn't really undestand the answer given: Test of significance of multiple proportions in two groups (it seems to suggest multiple chi-squared tests, is that it?)

Sorry for bothering you guys! Any help would be much appreciated!

  • $\begingroup$ Here is a reference on how to do Fisher exact tests with correspondingly exact multiplicity adjustments. pubmed.ncbi.nlm.nih.gov/18932134 $\endgroup$ Feb 9, 2021 at 16:32
  • $\begingroup$ @BigBendRegion thanks for the reference! So, in this case, there's no problem in making 5 different 2x2 contingency tables, applying Fisher exact test and correcting for multiplicity? $\endgroup$ Feb 9, 2021 at 20:40
  • $\begingroup$ Is $H_0$ supposed to be the null hypothesis? Or did you mean something else? $\endgroup$ Feb 9, 2021 at 20:54
  • $\begingroup$ @Luke_Blacck Yes, but you can take advantage of the discrete nature of the distributions to get much more power than simple Bonferroni. Depending on the nature of the distributions, it can be as dramatic as reducing the Bonferroni factor from 5 to 2 or so. See also tandfonline.com/doi/abs/10.1198/sbr.2010.09055 $\endgroup$ Feb 9, 2021 at 21:23
  • $\begingroup$ @ThePointer It is the null hypothesis, yes! Sorry for the bad formatting. $\endgroup$ Feb 9, 2021 at 22:47

1 Answer 1


When you perform "exact" permutation-based tests such as the Fisher exact test on the variables in a multi-sample multivariate data set, the discrete nature of sampling distributions of the test statistics can be "harvested" to provide extra power over Bonferroni types of adjustments. The correlation structure can be similarly "harvested" along with the discreteness, although the power gain from the discreteness is often much greater. The algorithm is in here:Westfall, P.H. and Troendle, J.F. (2008). Multiple Testing with Minimal Assumptions, Biometrical Journal 50, 745–755

Cases where the power gain is phenomenal are shown here: Westfall, P.H. (2011). Improving Power by Dichotomizing (Even Under Normality), Statistics in Biopharmaceutical Research 3, 353–362.

As it turns out, certain implementations of the methodology have been hard-coded in PROC MULTTEST of SAS since the early 1990s.

It is also worth noting that the late great John Tukey deserves some credit for this methodology based on his consultations with Merck.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.