# Post hoc test for Fisher's exact test (larger than 2x2)?

I have 2 categorical variables: Var1 - 3 levels, Var2- 5 levels (i.e., table is bigger than 2x2). I'm using SAS, and since I have multiple cells <5, I have run a Fisher's exact test with Monte Carlo estimation for the p value (code and output is below). I understand that if I want to determine which specific cells are not associated, I need to perform a post hoc test, but I haven't been able to determine if there is an appropriate post hoc test for Fisher's exact tests? Any help or insight would be very much appreciated.

• Are the variables ordinal or categorical? – AdamO Feb 16 at 23:36
• Both variables are categorical – Roaring Feb 16 at 23:37
• Why not just use a chi-square test of heterogeneity? – AdamO Feb 16 at 23:45
• I have some cells that are <5 and I was under the impression that you can't use chi square tests when the cell count is <5, but you can with Fisher's exact – Roaring Feb 17 at 0:14
• In R's implementation of 'chisq.test' you can usually simulate a valid P-value by using parameter 'sim=T' if you get an error message (triggered by small expected counts) that the P-value from the chi-squared statistic may not be valid. // My experience has been that this simulation often gives a P-value that approximates that of Fisher's exact test. // If using a chisq.test you can look at Pearson residuals; cells for which they have highest absolute values may point the way to sub-tables that may give significant ad hoc tests. – BruceET Feb 17 at 0:36

Comment continued:

Here is a contingency table (fake data) with some small counts; I can use it to illustrate some of the statements in my Comment. [I am using R.]

TBL = rbind(c( 3, 12,  3, 20, 10),
c(10, 15,  5, 10,  3))
TBL
[,1] [,2] [,3] [,4] [,5]
[1,]    3   12    3   20   10
[2,]   10   15    5   10    3


A chi-squared test on this table may have an incorrect P-value due to small counts:

chisq.test(TBL)

Pearson's Chi-squared test

data:  TBL
X-squared = 11.465, df = 4, p-value = 0.02181

Warning message:
In chisq.test(TBL) : Chi-squared approximation
may be incorrect


Specifically, expected counts in col 3 may be too small.

chisq.test(TBL)$exp [,1] [,2] [,3] [,4] [,5] [1,] 6.857143 14.24176 4.21978 15.82418 6.857143 [2,] 6.142857 12.75824 3.78022 14.17582 6.142857 Warning message: In chisq.test(TBL) : Chi-squared approximation may be incorrect  However, Pearson residuals point to columns 1 and 5 as possibly worth a closer look--provided the whole table turns out to be significant. chisq.test(TBL)$resi
[,1]       [,2]       [,3]      [,4]      [,5]
[1,] -1.472971 -0.5940281 -0.5937952  1.049740  1.200198
[2,]  1.556254  0.6276151  0.6273690 -1.109093 -1.268059
Warning message:
In chisq.test(TBL) : Chi-squared approximation
may be incorrect


We use simulation to obtain a more trustworthey P-value, leading to rejection of the null hypothesis.

chisq.test(TBL, sim = T)

Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)

data:  TBL
X-squared = 11.465, df = NA, p-value = 0.01849


Fisher's Exact Test on the $$2 \times 5$$ table gives roughly the same P-value.

fisher.test(TBL)

Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.02156
alternative hypothesis: two.sided


So we decide there are significant departures from independence (or homogeneity) and that this significance may be partly due to columns 1 and 5.

TB.15 = TBL[,c(1,5)];  TB.15
[,1] [,2]
[1,]    3   10
[2,]   10    3
chisq.test(TB.15, sim=T)

Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)

data:  TB.15
X-squared = 7.5385, df = NA, p-value = 0.01899


Alternatively, we might use Fisher's exact tests ad hoc to the Fisher test on the whole table (especially, if we did not need suggestions from Pearson residuals to help decide which ad hoc tests are of interest).

fisher.test(TB.15)

Fisher's Exact Test for Count Data

data:  TB.15
p-value = 0.01693
alternative hypothesis:
true odds ratio is not equal to 1
95 percent confidence interval:
0.009851234 0.720962703
sample estimates:
odds ratio
0.1011654


Notes: (1) If we wanted to look at several such ad hoc tests, we should use some method (such as Bonferroni's) to avoid 'false discovery'.

(2) It is best to choose either chi-squared tests (possibly with simulated P-values) or Fisher exact tests for use throughout the analysis--possibly stating a rationale for the choice. [It is not fair to run all the tests and choose to report the ones with the smaller P-values.]

(3) If your tables have a large proportion of expected counts below 5 (and especially below 3), and if SAS does not do simulated P-values for chi-squared tests on sparse tables, I would recommend Fisher exact tests. I don't know the context of your work. However, if it will be reviewed for publication or by government regulators, you may run into preference for strict observance of the rule that expected counts should exceed 5 with few minor exceptions.