# Interpreting Fisher's test and power analysis in R

Currently I want to check whether there is a significant difference between the data from two clusters with n1 = n2 = 6.

    cluster1 cluster2
av         0        1
cem        3        4
cl         2        4
cm         1        1
md         2        4
mtt        0        3
pf         1        1
pul        0        0
r          1        1
va         1        2
vl         1        5
vm         0        5
vp         0        0


As frequencies aren't bigger than 5 I decided to use Fisher's exact test, but I'm somewhat lost. Something that I noticed is how I put the data into Fisher's exact test.

For example fisher.test(df$cluster1, df$cluster2) delivers a p-value of 0.1192 for a two-sided Fisher's test while fisher.test(df) delivers a p-value of 0.7792. What is the correct usage? Secondly, I wonder how to conduct a compromise power analysis for Fisher's test. As my sample size is rather small using alpha = .05 is not a good idea. However, for calculating a power analysis I need the proportions but iirc the proportion is the ratio of the category of interest and the sample size.

Further, I wonder why I get a p-value of 0.7792 if it is the right p-value. As you can see I made a barplot for the distribution of thalamic nuclei in the two clusters. In my eyes, the barplot suggest that there is a significant difference between the two clusters, however, the p-value suggests there isn't.

Am I using the Fisher's test wrong? Is the p-value correct and my interpretation is simply wrong? And are there some other methods to analyse the distribution of my variables? • Given 13 levels, suppose a test gives you p =0.001, what conclusion can you get? – user158565 Aug 3 '19 at 17:40
• You're not very clear about the hypothesis of interest ("a difference" in the presence of many potential kinds of difference is not unambiguous). What are your trying to find out? Is your primary interest "does one cluster tend to have consistently higher/lower counts than the other?" (for which a different kind of chi-squared test could be suitable), or is it more "are the proportions across categories different for the two clusters" (for which you could use something like a $2\times k$ chi-squared) or is it something else? – Glen_b -Reinstate Monica Aug 4 '19 at 0:37
• [Note that the specific hypothesis should not be chosen after you have seen the data!] – Glen_b -Reinstate Monica Aug 4 '19 at 0:39
• Thanks for your comment. My primary interest is whether one cluster tends to have consistently higher/lower counts than the other. What kind of chi-squared test would be suitable? – Brain Damage Aug 4 '19 at 17:30
• Differences between clusters (each row). – BruceET Aug 4 '19 at 18:00

Two tests seem appropriate. Counts in Cluster 2 tend to be larger than counts in Cluster 1. I assume row counts are determined independently.

Sign test: Of the 13 row categories, Cluster 2 had higher counts than Cluster 1 for 8 categories, and and there is no difference for 5 categories. No row category had a smaller count in Cluster 2.

The sign test ignores the 5 categories with no difference. In 8 tosses of a fair coin the probability of getting all heads or all tails is $$(1/2)^7 = 0.0078 < 0.05$$ so there is a significant difference at the 5% level. Minitab output:

Sign test of median =  0.00000 versus ≠ 0.00000

N  Below  Equal  Above       P  Median
Dif  13      0      5      8  0.0078   1.000


Wilcoxon signed-rank test: Look at the 13 differences for Cluster 2 - Cluster 1. This test takes into account the sizes of the positive differences. The P-value is $$0.014 < 0.05,$$ again significant.

Wilcoxon Signed Rank Test

Test of median = 0.000000 versus median ≠ 0.000000

N for   Wilcoxon         Estimated
N   Test  Statistic      P     Median
Dif  13      8       36.0  0.014      1.000


Note: Neither Fisher's exact test nor (even with higher counts) a chi-squared test would be appropriate. The counts in Cluster 2 are consistently higher than the counts in Cluster 1 for the same category. But both of these tests are looking for inconsistency between the two clusters.

Here are Binomial counts, consistently about three times as large in B as in A:

set.seed(1234)
A = rbinom(13, 25, .3);  B = rbinom(13, 25, .9)
DTA = rbind(A,B);  DTA
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13]
A    5    8    8    8   10    8    3    6    8     8     9     8     6
B   20   23   21   23   24   24   24   23   23    24    25    24    21


I suppose you would have been happy with such a strong result in your experiment if you had more data. However, the hypothesis of the chi-squared test for homogeneity is (appropriately) not rejected.

chisq.test(DTA)

Pearson's Chi-squared test

data:  DTA
X-squared = 3.9673, df = 12, p-value = 0.984

• The sign test sounds promising. How did you transform the data to apply it on a sign test? – Brain Damage Aug 4 '19 at 17:48