How do you determine the significance of individual deviations in categories?

Question

I have observations from human observers that were intended to place subjects into one of four categories (A-D). In fact, there are 100 subjects in each category. So in an ideal world there should be "100" in each category.

            |  A  |  B  |  C  |  D
Observation | 87  | 110 | 101 | 102
'Ideal'     | 100 | 100 | 100 | 100

If I wanted to know whether the observations as a whole are statistically significantly different from the ideal case, I would do a Chi-Square test.

But which test can tell me whether the observations for a particular category differ significantly? For example, is the value "87" (category A) significantly different?

Answer 1

Chi-squared test. It seems you want to see whether results are consistent with 1/4 of the subjects being put into each of 4 categories. (If you were vetting a die to see if it is fair, you might roll it 600 times to see if you got nearly 100 counts for each face.)

In R, the chi-squared test goes as shown below. It shows no evidence to reject the null hypothesis that categories are equally likely: P-value far above 0.05.

chisq.test(c(87,110,101,102))

        Chi-squared test for given probabilities

data:  c(87, 110, 101, 102)
X-squared = 2.74, df = 3, p-value = 0.4335

Notes: (a) Given this result, it is not appropriate to check whether each individual category has counts consistent with 1/4.

(b) If no probability vector is supplied, then the 'given probabilities' are assumed to be equal.

(c) The 'expected counts' used in this chi-squared test are as follows. Strictly speaking, they are not part of your data table.

chisq.test(c(87,110,101,102))$exp
[1] 100 100 100 100

Binomial test. If you had doubts before seeing the data whether Category A would be chosen 1/4 of the time, you could have done a binomial test just for Category A, as shown below. Again, the result is not significant.

binom.test(87,400, 1/4)

        Exact binomial test

data:  87 and 400
number of successes = 87, number of trials = 400, 
  p-value = 0.1487
alternative hypothesis: 
  true probability of success is not equal to 0.25
95 percent confidence interval:
 0.1780405 0.2611969
sample estimates:
probability of success 
                0.2175 

However, to be clear, it is not appropriate to do the binomial test as an 'ad hoc' test to the non-significant chi-squared test.

Simulation as 'reality check'. Below are four simulated versions of your experiment, in which it is known that categories 1 through 4 are chosen with equal probability. From this brief simulation, it seems that counts as low as 87 in one of the four categories are not especially rare.

set.seed(2020)
x = sample(1:4, 400, rep=T); table(x)
x
  1   2   3   4 
111  83 107  99 
x = sample(1:4, 400, rep=T); table(x)
x
  1   2   3   4 
 93 109  96 102 
x = sample(1:4, 400, rep=T); table(x)
x
  1   2   3   4 
103 117  94  86 
x = sample(1:4, 400, rep=T); table(x)
x
  1   2   3   4 
100 101 100  99 

In a simulation of 100,000 such experiments, the smallest category count was $\le 87$ more than a quarter of the time.

set.seed(701)
min.ct = replicate(10^5, 
                   min(tabulate(sample(1:4, 400, rep=T))))
mean(min.ct <= 87)
[1] 0.28186