Test if differences between frequencies is significant

Question

I have the following frequency table:

35  0   4   3   7   6   5   4
39  1   9   6   7   7   6   8
36  0   7   10  11  11  10  16
41  0   9   8   8   7   6   7
41  0   8   9   10  9   12  11
55  2   12  9   11  12  11  13
55  1   10  10  11  10  12  11
47  1   14  8   12  15  12  12
45  1   10  11  10  10  9   18
56  0   13  16  12  12  12  11

The Kruskal-Wallis ANOVA test returns:

Source      SS      df     MS     Chi-sq   Prob>Chi-sq  
Columns  25306.8    7   3615.26   47.16    5.18783e-008  
Error     17083.2   72   237.27
Total     42390     79

According to a multiple comparison of mean ranks:

• Six groups of mean significantly different from group 1 (column 1)

• Six groups of mean significantly different from group 2 (column 2)

---

Now the Kruskal-Wallis and multiple comparison tests make sense, however the Chi Square Test returns a chi square value of 31.377 and a p-value of 0.9997, which leads us to accept the null hypothesis that the frequencies are independent. I understand that an assumption of ANOVA is independence, but...

I want to see test if the frequencies are statistically independent, was the Kruskal-Wallis and multiple comparison tests the correct methodology? Note: I am not trying to be subjective, but for a given set of frequencies, how do you test that the differences between groups are significant?

Answer 1

Based on your additional explanation in the comments, it appears that you have 8 groups (each corresponding to a column) and a continuous outcome variable that you grouped into 10 bins (each bin corresponding to a row). Note that it also implies that the rows are ordered with later rows implying larger values.

First of all, if you do have the underlying continuous variable, then do not bin it - just use Kruskall-Wallis or ANOVA to compare the groups.

Assuming that the binning is unavoidable, you can still use a Kruskall-Wallis test, but not on the frequencies as you have apparently done it. Your current KW inference just tells you that you have more data in some groups as compared to others. The actual observations in this case are the row numbers (1 through 10), and the values in the table are just the frequencies of occurrences. Most statistical software has an option of specifying these as "weights" or "frequencies".

The chi-square test can be used on the frequencies, however if the rows are ordered it might have much lower power compared to the Kruskall-Wallis test to actually detect differences, since it completely ignores the ordering of the rows. Thus even though its results are valid, I would not recommend using these due to the loss of power.

Answer 2

Was going to leave this as a comment, but it was getting too long...

While the chi-square statistic may not be significant, its numerical value is large. You can interpret $\frac{1}{2}\chi^{2}$ as an approximate log-likelihood ratio against the best alternative in the Bernoulli class. So the data supports the best alternative (Observed=Expected) to the independence hypothesis by a factor of $\exp\left(\frac{1}{2}\chi^{2}\right)=6,507,722$. So you are justified in looking for a better hypothesis, as your intuition suggests.

The problem is that with $10*8-1=79$ degrees of freedom, there are a huge number of alternatives, so you are at risk of finding spurious relationships in your (i.e. "fitting the noise"). So you basically need some good prior information to back up this better alternative.

Column 1 and Column 2 are clearly different in terms of how they are represented in the data as your test appears to indicate (column1 has many more, and column2 has almost zero observations). If you separate off these, the Chi-square for the remaining $6$ columns is now just $15.83$ for an approximate likelihood ratio of $2,741$. However, this could well be just an artifact of the sampling scheme.