Let's say we do a poll/survey for $n=250$ people, and we study a variable in a Likert scale with 7 possible values (-3 to +3):

Person    Subgroup     Question1
1         B            -3 (strongly disagree)
2         C            0 (neutral)
3         A            2 (agree)
4         A            1 (slightly agree)
5         E            -2 (disagree)
...       ...          ...
250       A            3 (strongly agree)

We want to evaluate/test the hypothesis:

"People from subgroup A tend to have a significantly higher agreement with Question1 (i.e. higher value for Question1 in Likert scale) than the general population."

This is an unprecise "handwaving" formulation. Question:

  1. How should we reformulate this hypothesis more precisely to be able to use usual statistical hypothesis testing tools, and be able to reject or not reject the hypothesis with $p = 0.05$, $p=0.01$, etc.?

  2. Which common statistical hypothesis testing tools can be used in such situation?

    I know the $\chi^2$ fit test to see if an observation matches a fixed distribution (goodness of fit); or also the $\chi^2$ test for homogeneity or independance; but none of these seem to apply here.
    Or does it apply?
    Or should one use other tests? I often hear about t-test, Fisher, Kruskal-Wallis, ANOVA, Dunn, so I was curious which one to use here.


First, you should compare Group A with other Groups (are they B through E?) combined. Do not compare Group A with all subjects (including Group A itself).

As you say, you could do this using a chi-squared test, as illustrated below:

Suppose you have data roughly as below. [In sample, parameter p gives proportions for the seven categories, which need not add to 1.]

a =   sample(-3:3,  50, rep=T, p = c(1,2,3,3,3,4,5))
b.e = sample(-3:3, 217, rep=T, p = c(1,2,3,4,5,4,3))
-3 -2 -1  0  1  2  3 
 3  4 10  6  5  6 16 
-3 -2 -1  0  1  2  3 
11 18 24 46 46 32 40 

TAB = rbind(c(3,4,10,5,5,6,16), c(11,18,24,46,46,32,40))
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    3    4   10    5    5    6   16
[2,]   11   18   24   46   46   32   40

A chi-squared test in R gives a highly significant P-value, but with a warning message that it may not be accurate.

chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 11.873, df = 6, p-value = 0.06487

Warning message:
In chisq.test(TAB, cor = F) : 
  Chi-squared approximation may be incorrect

This message is shown because a couple of the expected counts for the chi-squared test are smaller than 5:

chisq.test(TAB, cor=F)$exp
          [,1]      [,2]      [,3]      [,4]      [,5] [,6]     [,7]
[1,]  2.578947  4.052632  6.263158  9.394737  9.394737    7 10.31579
[2,] 11.421053 17.947368 27.736842 41.605263 41.605263   31 45.68421

Some statisticians would use the P-value if only a few expected counts are smaller than 5, but not if any are below 3.

In R, it is possible to simulate a useful P-value, as shown below:

chisq.test(TAB, cor=F, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 11.873, df = NA, p-value = 0.06297

So it seems that the the null hypothesis could be rejected at the 10% level, but not at the 5% level.

Of course, your data may be different from my simulated data (either more subjects in Group A, or a more even distribution of opinions).

I wanted you to be aware that one often runs into trouble with low expected values if you subset the data. Also, that the option to simulate in R's chisq.test is sometimes helpful.

Note: In any case, when the full $5\times 7$ table for 5 groups and 7 categories, with 35 cells altogether, uses only 267 subjects, that's an average of 7 or 8 counts per cell, giving a high risk of expected counts that are too low.

Here is are data simulated with about twice as many subjects and the same proportions. There is a warning message about one expected value a little below 5, but a highly significant result with a simulated P-value. (Without simulation the possibly erroneous P-value is about 0.0001.)

[Also, not that using category numbers 1 to 7 instead of -3 to 3, facilitates making the contingency table; a 'quirk' of tabulate.]

a =   sample(1:7,  100, rep=T, p = c(1,2,3,3,3,4,5))
b.e = sample(1:7, 400, rep=T, p = c(1,2,3,4,5,4,3))
TBL = rbind(tabulate(a), tabulate(b.e))  TBL
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    6    6   20   16    9   12   31
[2,]   17   39   62   86   74   66   56
chisq.test(TBL, cor=F)$exp
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]  4.6    9 16.4 20.4 16.6 15.6 17.4
[2,] 18.4   36 65.6 81.6 66.4 62.4 69.6

chisq.test(TBL, cor=F, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 22.632, df = NA, p-value = 0.0004998

Then you might do an ad hoc chi-squared test on the $2\times 2$ table (combining the lowest six categories) to target your exact issue of interest.

Tbl = rbind(c(69, 31), c(344, 45))
[1] 3.675252e-06
  • $\begingroup$ Thank you very much @BruceET for your detailed answer. What kind of Chi squared test is it, is it a goodness of fit one, a homogeneity one, an independance one, or another one? $\endgroup$ – Basj Apr 9 at 5:53
  • $\begingroup$ Also, what is exactly the null hypothesis that is being tested with this method? I don't see exactly where is used the part of the hypothesis "agreement is higher for group A" in this method. Thanks in advance! $\endgroup$ – Basj Apr 9 at 5:59
  • $\begingroup$ The null hypothesis for this chi-squared test is that the two categorical variables (A vs. all other groups) have the same distributions. Part of doing the chi-squared test is to find expected counts $E_{ij}$ for each of the $2 \times 7$ cells. That computation uses the null hypothesis. Then the chi-squared statistic uses the formula $Q = \sum_i\sum_j \frac{(X_{ij}-E_{ij})^2}{E_{ij}}$ to compare observed counts $X_{ij}$ with corresponding expected counts. [There are 14 terms in the sum.] Under $H_0,$ $Q \stackrel{aprx}{\sim}\mathsf{Chisq}(df=6),$ provided expected counts are large enough. $\endgroup$ – BruceET Apr 9 at 7:18
  • $\begingroup$ Thanks a lot @BruceET. Maybe you can edit to include this precise statement of the null hypothesis inside the question (so that everything is crystal clear) + your last remarks and the sum $Q$, or do you prefer that I do it? $\endgroup$ – Basj Apr 9 at 7:52
  • 1
    $\begingroup$ Last thing: is it possible/common in such surveys to do a test not on "H0: The two categorical variables have the same distribution" but rather "H0: Class A has a higher agreement value than the other classes"? Not sure what kind of test to do the latter? $\endgroup$ – Basj Apr 9 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.