Evaluate the increase of a variable (in Likert scale) for a subgroup of the population in a poll

Question

Let's say we do a poll/survey for $n=250$ people, and we study a variable in a Likert scale with 7 possible values (-3 to +3):

Person    Subgroup     Question1
1         B            -3 (strongly disagree)
2         C            0 (neutral)
3         A            2 (agree)
4         A            1 (slightly agree)
5         E            -2 (disagree)
...       ...          ...
250       A            3 (strongly agree)

We want to evaluate/test the hypothesis:

"People from subgroup A tend to have a significantly higher agreement with Question1 (i.e. higher value for Question1 in Likert scale) than the general population."

This is an unprecise "handwaving" formulation. Question:

1. How should we reformulate this hypothesis more precisely to be able to use usual statistical hypothesis testing tools, and be able to reject or not reject the hypothesis with $p = 0.05$, $p=0.01$, etc.?

2. Which common statistical hypothesis testing tools can be used in such situation?

   I know the $\chi^2$ fit test to see if an observation matches a fixed distribution (goodness of fit); or also the $\chi^2$ test for homogeneity or independance; but none of these seem to apply here.
   Or does it apply?
   Or should one use other tests? I often hear about t-test, Fisher, Kruskal-Wallis, ANOVA, Dunn, so I was curious which one to use here.

Answer 1

First, you should compare Group A with other Groups (are they B through E?) combined. Do not compare Group A with all subjects (including Group A itself).

As you say, you could do this using a chi-squared test, as illustrated below:

Suppose you have data roughly as below. [In sample, parameter p gives proportions for the seven categories, which need not add to 1.]

set.seed(408)
a =   sample(-3:3,  50, rep=T, p = c(1,2,3,3,3,4,5))
b.e = sample(-3:3, 217, rep=T, p = c(1,2,3,4,5,4,3))
table(a)
a
-3 -2 -1  0  1  2  3 
 3  4 10  6  5  6 16 
table(b.e)
b.e
-3 -2 -1  0  1  2  3 
11 18 24 46 46 32 40 

TAB = rbind(c(3,4,10,5,5,6,16), c(11,18,24,46,46,32,40))
TAB
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    3    4   10    5    5    6   16
[2,]   11   18   24   46   46   32   40

A chi-squared test in R gives a highly significant P-value, but with a warning message that it may not be accurate.

chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 11.873, df = 6, p-value = 0.06487

Warning message:
In chisq.test(TAB, cor = F) : 
  Chi-squared approximation may be incorrect

This message is shown because a couple of the expected counts for the chi-squared test are smaller than 5:

chisq.test(TAB, cor=F)$exp
          [,1]      [,2]      [,3]      [,4]      [,5] [,6]     [,7]
[1,]  2.578947  4.052632  6.263158  9.394737  9.394737    7 10.31579
[2,] 11.421053 17.947368 27.736842 41.605263 41.605263   31 45.68421

Some statisticians would use the P-value if only a few expected counts are smaller than 5, but not if any are below 3.

In R, it is possible to simulate a useful P-value, as shown below:

chisq.test(TAB, cor=F, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 11.873, df = NA, p-value = 0.06297

So it seems that the the null hypothesis could be rejected at the 10% level, but not at the 5% level.

Of course, your data may be different from my simulated data (either more subjects in Group A, or a more even distribution of opinions).

I wanted you to be aware that one often runs into trouble with low expected values if you subset the data. Also, that the option to simulate in R's chisq.test is sometimes helpful.

---

Note: In any case, when the full $5\times 7$ table for 5 groups and 7 categories, with 35 cells altogether, uses only 267 subjects, that's an average of 7 or 8 counts per cell, giving a high risk of expected counts that are too low.

Here is are data simulated with about twice as many subjects and the same proportions. There is a warning message about one expected value a little below 5, but a highly significant result with a simulated P-value. (Without simulation the possibly erroneous P-value is about 0.0001.)

[Also, not that using category numbers 1 to 7 instead of -3 to 3, facilitates making the contingency table; a 'quirk' of tabulate.]

set.seed(408)
a =   sample(1:7,  100, rep=T, p = c(1,2,3,3,3,4,5))
b.e = sample(1:7, 400, rep=T, p = c(1,2,3,4,5,4,3))
TBL = rbind(tabulate(a), tabulate(b.e))  TBL
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    6    6   20   16    9   12   31
[2,]   17   39   62   86   74   66   56
chisq.test(TBL, cor=F)$exp
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]  4.6    9 16.4 20.4 16.6 15.6 17.4
[2,] 18.4   36 65.6 81.6 66.4 62.4 69.6

chisq.test(TBL, cor=F, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 22.632, df = NA, p-value = 0.0004998

Then you might do an ad hoc chi-squared test on the $2\times 2$ table (combining the lowest six categories) to target your exact issue of interest.

Tbl = rbind(c(69, 31), c(344, 45))
chisq.test(Tbl)$p.val
[1] 3.675252e-06