I have two groups of categorical data as follows, which I observed from one population by two different methods:

    Group1   Group2
T1    60        68
T2    60        53  
T3    70        72
T4    60        84  

Now, I am looking for the difference between this two samples by deffinning the null hypothesis (H0) as "there is no difference between these two samples". So I want to reject the H0 if there is a difference between them and say that the results are statistically different. My question is, what is the best statistical test and hypothesis for this categorical data? Is Chi-Square test correct test for this problem? I applied chi-Square test on this data, but I got the following results:

X^2=3.587  P-value=0.309 , variables are independent in alpha=0.05

Based on the null-hypothesis, I should have rejected the H0 and concluded that the two samples are different.
In this case, the null-hypotheis is "there is a difference between the two groups" and I failed to reject it. So I got this result. I am not sure failing to reject the null-hypothesis is a positive result in this case. Does anyone know, how I can handle this matter, by diffinning the null-hypothesis and applying a correct statistical test to show the difference?

  • $\begingroup$ Plainly there's a different between the samples. You don't use hypothesis tests to discover that (look at the numbers ... they're different!). Hypothesis tests have a different purpose (generally, to infer something about populations) $\endgroup$
    – Glen_b
    Aug 21, 2018 at 0:46

1 Answer 1


Because of the round numbers in the first column (group), I wonder if all eight numbers are counts of subjects randomly chosen from two populations.

If you randomly assigned 250 subjects to one Method (col 1) and 277 subjects to the other Method (col 2), and then later each subject was found to be in one of four categories (T1, ..., T4), then it would be appropriate to perform a chi-squared test of independence on these data, as follows:

DATA = matrix(c(60,60,70,60,  68, 53, 72, 84), nrow=4)
     [,1] [,2]
[1,]   60   68
[2,]   60   53
[3,]   70   72
[4,]   60   84

The chi-squared statistic is $$Q = \sum_{i=1}^4 \sum_{j=1}^2 \frac{(X_{ij}-E_{ij})^2}{E_{ij}},$$ where $E_{ij}$ are expected counts and $X_{ij}$ are the observed counts in the matrix above.

Expected counts are obtained by taking "(row total)(col total)/(grand total)" for each cell in the matrix; for example $E_{11} = \frac{128\,\cdot\, 250}{527} = 60.72106.$ Here is a matrix of all six expected values:

         [,1]     [,2]
[1,] 60.72106 67.27894
[2,] 53.60531 59.39469
[3,] 67.36243 74.63757
[4,] 68.31120 75.68880

Then under $H_0,$ we have $Q \stackrel{aprx}{\sim}\mathsf{Chisq}(\nu = 3),$ where the degrees of freedom are found as $\nu = (r-1)(c-1) = 3(1) = 3.$ The approximation is useful if all six $E_{ij}$'s are at least 5.

In R the test can be performed as follows:


        Pearson's Chi-squared test

data:  DATA
X-squared = 3.5879, df = 3, p-value = 0.3095

So there is no evidence at the 5% level that the proportions of T1, T2, T3 and T4 differ across Methods 1 and 2. (You would need $Q > 7.815$ instead of $Q = 3.588$ for a result significant at the 5% level.) More succinctly, Method does not appear to have made a significant difference.

qchisq(.95, 3)
[1] 7.814728

Strictly as a 'thought-expteriment', consider this: If you had exactly the same relative proportions among expected and observed counts, but with more data, you might have had a significant result. For example, let's imagine you had three times as many subjects with similar proportional resuls:

     [,1] [,2]
[1,]  180  204
[2,]  180  159
[3,]  210  216
[4,]  180  252

        Pearson's Chi-squared test

data:  DATA * 3
X-squared = 10.764, df = 3, p-value = 0.01307

Notice that if you made bar charts for DATA and for DTA3, they would look almost exactly the same (except possibly for labels on axes). Yet one would show data with no significant difference and one would show highly significant data.

This illustrates that bar charts alone are almost useless in judging the significance of categorical data. Every bar chart for categorical data should have a footnote or caption showing the P-value of the corresponding chi-squared test.

In a chi-squared test the degrees of freedom are determined by the numbers of levels of the categories, yet sample size is reflected in the value of $Q$ and in the P-value in ways that may not be intuitively obvious.

As a simpler example of a similar comparison: If you throw a coin 10 times and get 7 heads, that is not convincing evidence the coin is biased; but if you get 697 heads in 1000 tosses, that would be strong evidence of bias.

  • $\begingroup$ Tnx for the complete explanation. To response to your question, the samples are same. It means we had 160 people in each sample, and we implemented two different algorithms on the same people and got different numbers in different categories (classification problem). Indeed, the numbers are the percentage of correctly classification each category (T1,..,T4). E.g.,, with method1 we obtained 60% correctly classification for T1, while 68% correctly classification by method2, and so on. $\endgroup$ Aug 21, 2018 at 7:38
  • 1
    $\begingroup$ For chi-squared computation, you should use counts not percentages. $\endgroup$
    – BruceET
    Aug 21, 2018 at 7:46
  • $\begingroup$ Is it fine if I consider the number of correctly classification in this case (as you said) taking into account the whole numbers will be different in the two samples. $\endgroup$ Aug 21, 2018 at 7:57
  • $\begingroup$ Not sure I have the model right. Let's try this, and I'll think about it again tomorrow: Please try describing the procedure, step by step. Suppose I'm one of the 160 people. What do you do to me and what response(s) do I give you, and when? $\endgroup$
    – BruceET
    Aug 21, 2018 at 8:00
  • $\begingroup$ Based on the comment by @nahidkhosh to this answer, it sounds like a chi-square test of association isn't the right approach, at least as the data are set up. It sounds like logistic regression would be an approach, or perhaps Cochran–Mantel–Haenszel test. $\endgroup$ Aug 22, 2018 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.