# How to test whether different observers are getting the same count result for different species?

I have a set of data with over ten 'observers' and over ten 'species'. The below image is an example of how the data is set out in the table, although I also have it in a three column format of 'species', 'observer' and 'count'. The numbers are all just count values. My null hypothesis is that there will be no difference in count for a species when observer changes. However each species has a different, unknown expected count. e.g. if the null is correct every observer of species A should record the same count number but for species B those counts will differ from A but not differ as observer changes. I want to know whether there is a statistical test that is appropriate to apply to this data to find out whether or not observers differ in their ability to get these different expected counts for each experiment. The big problem I'm having is that I don't know the expected values, but I want to statistically show whether or not observers differ in their counts, but I do know that each species should have a different expected value. If you also have the basic R code that I can apply my own data set too that would be great but mostly I'm just looking to figure out what test I should be doing and I should be able to take it from there!

• I don't think you really need a formal test to see that the observers are giving different results. (In particular, 'Observer 2' doesn't seem to be seeing what the others do.) If there were a doubt, a chi-sq test of homogeneity would be appropriate: chisq.test(TBL) where TBL is your table of counts would be the procedure in R. I got P-val almost 0, overwhelmingly rejecting $H_9$ that counts for species are homogeneous across Observers. Jul 9, 2020 at 17:41
• Thanks for replying. I don't think I made it clear enough but the above table is just an example set, not my actual data. My actual table is far too big to do a visual assessment. I will look into applying Chi of homogeneity, thank you. Jul 10, 2020 at 22:50
• In that case I'll show my run of the chi-squared test in R. Jul 10, 2020 at 22:55

I'm using rows of the table for observers, because we are looking for homogeneity among them. (Ad hoc comparisons may be easier using observes as rows.) Computations in R.

x1 = c(50,35,0,23)
x2 = c(0,0,1,2)
x3 = c(12,23,13,24)
x4 = c(23,45,2,31)
TAB = rbind(x1,x2,x3,x4)
chsq.out=chisq.test(TAB)
Warning message:           # Look at expected counts below
In chisq.test(TAB) :
Chi-squared approximation may be incorrect

chsq.out

Pearson's Chi-squared test

data:  TAB
X-squared = 56.922, df = 9, p-value = 5.231e-09


The P-value near 0 indicates a highly significant result, leading to rejection of the null hypothesis $$H_0.$$ The number of degrees of freedom is $$\nu = (r-1)(c-1) = 3(3) = 9,$$ for your $$4 \times 4$$ table. The P-value is the probability, under $$H_0,$$ of a value greater than 56.922 for a chi-squared random variable with $$\nu = 9:$$

1 - pchisq(56.922, 9)
[1] 5.230288e-09

chsq.out$obs # echo of TAB [,1] [,2] [,3] [,4] x1 50 35 0 23 x2 0 0 1 2 x3 12 23 13 24 x4 23 45 2 31 chsq.out$exp               # note cells with <4 exp'd count
[,1]      [,2]      [,3]       [,4]
x1 32.3239437 39.169014 6.0845070 30.4225352
x2  0.8978873  1.088028 0.1690141  0.8450704
x3 21.5492958 26.112676 4.0563380 20.2816901
x4 30.2288732 36.630282 5.6901408 28.4507042

chsq.out$res [,1] [,2] [,3] [,4] x1 3.1090178 -0.6661347 -2.466679 -1.3457196 x2 -0.9475692 -1.0430859 2.021307 1.2563454 x3 -2.0570972 -0.6091275 4.440668 0.8256453 x4 -1.3147998 1.3828998 -1.546968 0.4779404  There are 16 cells in the contingency table, thus 16 'contributions' $$\frac{(X_{ij} - E_{ij})^2}{E_{ij}},$$ for $$i,j, = 1,2,3,4.$$ Here $$X_{ij}$$ are observer counts (integers), $$E_{ij}$$ are expected counts as determined from row and column totals using the null hypothesis of homogeneity (do not round severely). The sum of these 16 'contributions' is the chi-squared statistic. 'Signed square roots' of the contributions are called Pearson Residuals. The sum of squares of the residuals is the chi-squared statistic $$56.922.$$ sum(chsq.out$res^2)
[1] 56.92175


Thus, the residuals with the larger expected values (especially those above 2) may point to differences between observed and expected counts leading to a significant result and rejection of the null hypothesis. Here 'Obsever 2' seems to have given different results than the other observers.

Ad hoc comparisons may help to understand the significant result. For example, we can compare Observers 1 and 2, with an ad hoc chi-squared test:

chisq.test(cbind(x1,x2))\$p.val
[1] 6.447593e-09
Warning message:
In chisq.test(cbind(x1, x2)) :
Chi-squared approximation may be incorrect


You should use some method of avoiding 'false discovery' such as the Bonferroni method if you do several ad hoc tests. The idea is to reject only at levels smaller than 5%. (This is not an issue with our very small P-value for the overall chi-squared test.) Also, I don't expect your larger table will lead to expected counts small enough to trigger error messages.

Addendum per Comments: In the following (fake) example, Observer 4 does not see as many animals of any species as do the other three observers, but his/her counts are proportionately lower across all species. (The proportionality is actually 'too good to be true', as suggested by the extremely high P-value. It's almost as if Observer 4 just looked at notes of Observer 1, and reported about 80% as many sightings across all species. "When the P-value is tiny, doubt the null hypothesis; when the P-value is almost 1, give a second look at the model and the data.")

y1 = c(50,25,10,50)
y2 = c(47,28,12,52)
y3 = c(55,27,13,55)
y4 = c(41,20, 8,38)
TBL = cbind(y1,y2,y3,y4)
chisq.test(TBL)

Pearson's Chi-squared test

data:  TBL
X-squared = 0.87592, df = 9, p-value = 0.9997

• Thank you for the walk through, it's really helpful. Probably fairly obviously, I'm pretty rusty with statistics at the moment and trying to remember how it all works. Just to double check with what you've done above, is it the initial chi sq test that's showing somewhere within the rows an observer row is recording significantly different values in a species column to other rows for that corresponding species column, and then the ad hoc testing is narrowing down which observers specifically have the strongest differences? Jul 11, 2020 at 20:26
• I'm trying to make sure I don't report observer differences that are actually due to the fact that you'd expect different counts for each species column within single observers anyway. I'm just trying to clarify if the test is showing differences across the whole table, or just whether there are differences within individual species columns. Thank you for the help! Jul 11, 2020 at 20:26
• It's OK if some species are more prevalent than others. But for homogeneity among observers, their counts should all reflect roughly that pattern of prevalence (taking randomness of observation into account). // See new (fake) example in Addendum to my Answer. Jul 11, 2020 at 21:01
• Thank you for your help, it's really appreciated. I have another table I'm doing the same with but in this one over half of the expected values are below 5. I read that chi squared wasn't appropriate for this and Fisher's exact test should be used instead, but I don't really understand why. Are you able to explain it? Jul 23, 2020 at 9:54
• "Chi-squared" statistic doesn't have chi-squared dist'n when so many expected counts are below 5. Combine categories to get larger expected countsz, if feasible. Some authors say a couple of counts < 3 OK if rest are > 5. Jul 23, 2020 at 14:35