Comparing multiple sample proportions between two groups

Question

Hi all and thanks for taking the time to read this question. I'll try to be as clear as possible!

The data: I have one sample of patients which was divided in two groups based on a binary variable for a condition - Condition A and Condition B. Gorup A contains approx. 500 patients, while B contains approx. 170. The patients records have a categorical variable with 5 different (mutually exclusive) outcomes, say $[o_1,o_2,o_3,o_4,o_5]$.

The problem: I want to compare the proportions of each of those outcomes between the groups, i.e., I want to know if the proportion of outcome o1 is different between groups A and B, AND I want to know if the proportion of outcome $o_2$ is different between A and B, and so on. There would be multiple $H_0$'s in this case, in the line of: "Outcome $o_1$ is the same between groups".

Additional details: Some of the outcomes have very small observed proportions (e.g. 1 in 170, 7 in 500).

My thoughts: I've compared groups like this before, but never had to determine which outcome was different, only if the overall outcome proportions were dependent on group (Fisher's test and Chi-Squared test, etc). The groups do form contingency tables, since a patient cannot have more than one outcome. Is there any specific test for this situation? Or should I use something like a Chi-Squared test for each outcome and then perform a multiplicity adjustment? I found this thread, which seems to be similar, however I couldn't really undestand the answer given: Test of significance of multiple proportions in two groups (it seems to suggest multiple chi-squared tests, is that it?)

Sorry for bothering you guys! Any help would be much appreciated!

Answer 1

When you perform "exact" permutation-based tests such as the Fisher exact test on the variables in a multi-sample multivariate data set, the discrete nature of sampling distributions of the test statistics can be "harvested" to provide extra power over Bonferroni types of adjustments. The correlation structure can be similarly "harvested" along with the discreteness, although the power gain from the discreteness is often much greater. The algorithm is in here:Westfall, P.H. and Troendle, J.F. (2008). Multiple Testing with Minimal Assumptions, Biometrical Journal 50, 745–755

Cases where the power gain is phenomenal are shown here: Westfall, P.H. (2011). Improving Power by Dichotomizing (Even Under Normality), Statistics in Biopharmaceutical Research 3, 353–362.

As it turns out, certain implementations of the methodology have been hard-coded in PROC MULTTEST of SAS since the early 1990s.

It is also worth noting that the late great John Tukey deserves some credit for this methodology based on his consultations with Merck.