Multiple Hypothesis Tests - Chi Square

Question

I want to write a method to test multiple hypothesese for a pair of schools . I want to consider all possible pairs of words (Research Thesis Proposal AI Analytics), and test the hypothesis that the words counts differ significantly across the two schools, using the specified alpha (0.05) threshold.

Only need to conduct tests on words that have non-zero values for both schools. I.e., every row and column in the contingency table should sum to >0.

Finally, want to return a tuple with the (i) the total number of tests conducted, and (ii) the number of significant tests.

df:

Names        Research Thesis   Proposal   AI   Analytics Data
TAMU           54      0        0         6       5       0
uiuc           33      43       5         0       76      81
USC             4       1       0         7       21      4
UT Austin      22      31       0         0       55      0
UCLA           55       6       7         9       11      12

def school_term_hypotheses(filename,college1, college2, alpha):
   
   df=pd.read_csv(filename)
   df=df[(df['Name'] == college1) | (df['Name'] == college2)]
   df=df.loc[:, df.ne(0).all()]
   df=df.set_index('Unnamed: 0')
   #chi,p=chi2_contingency(df)[:2]
   #return(p)

school_term_hypotheses("test.csv", 'TAMU','UT Austin' 0.05)

I am clueless what to do after getting a df with non zero values. need some help figuring how do I test multiple hypothesese.

Answer 1

With little trouble you can use the entire contingency table to test whether the distribution of the six words is homogeneous across the five schools.

MAT = matrix( c(54,  0,  0,  6,  5,  0,
                33, 43,  5,  0, 76, 81,
                 4,  1,  0,  7, 21,  4,
                22, 31,  0,  0, 55,  0,
                55,  6,  7,  9, 11, 12), byrow=T, nrow=5)
 MAT
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]   54    0    0    6    5    0
[2,]   33   43    5    0   76   81
[3,]    4    1    0    7   21    4
[4,]   22   31    0    0   55    0
[5,]   55    6    7    9   11   12

chisq.test(MAT)

        Pearson's Chi-squared test

data:  MAT
X-squared = 318.4, df = 20, p-value < 2.2e-16

Warning message:
In chisq.test(MAT) : 
 Chi-squared approximation may be incorrect

The warning message arises because several of the expected counts in the columns for 'Proposal' and 'AI' are below 5 (see note at end). Fortunately, the implementation of chisq.test in R can simulate a useful P-value---which happens to be much smaller than 5%. So the null hypothesis of homogeneity is strongly rejected.

chisq.test(MAT, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  MAT
X-squared = 318.4, df = NA, p-value = 0.0004998

If you wanted to test homogeneity for TAMU and UCLA, you could simply select the first and fifth rows of MAT, finding a significant difference in the distribution of the six words.

chisq.test(MAT[c(1,5), ], sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  MAT[c(1, 5), ]
X-squared = 21.398, df = NA, p-value = 0.0004998

For a test of TAMU against USC, it is probably best to leave out the third word-column also (because it has two 0's). Again the null hypothesis of homogeneity is rejected.

MAT[c(1,3),c(1:2,4:6)]
     [,1] [,2] [,3] [,4] [,5]
[1,]   54    0    6    5    0
[2,]    4    1    7   21    4

chisq.test(MAT[c(1,3),c(1:2,4:6)], sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  MAT[c(1, 3), c(1:2, 4:6)]
X-squared = 54.443, df = NA, p-value = 0.0004998

According to the Bonferroni method of avoiding false discovery from repeated tests on the same data, if you are doing $k$ of these ad hoc 2-school comparisons, you should test at level $(5/k)\%.$

Note: Here are the expected counts under the null hypothesis of homogeneity for the first use of chisq.test above:

chisq.test(MAT)$exp
         [,1]      [,2]     [,3]     [,4]     [,5]     [,6]
[1,] 19.92701  9.607664 1.423358 2.609489 19.92701 11.50547
[2,] 72.96350 35.178832 5.211679 9.554745 72.96350 42.12774
[3,] 11.34307  5.468978 0.810219 1.485401 11.34307  6.54927
[4,] 33.10949 15.963504 2.364964 4.335766 33.10949 19.11679
[5,] 30.65693 14.781022 2.189781 4.014599 30.65693 17.70073