"Counts" in contingency tables: Trying to understand misunderstandings with chi squared tests

Question

I am attempting to understand what I think is a misunderstanding of contingency table analysis among some biologists. In particular, I see the use of contingency tables for the analysis of numeric count data when studies lack replication. This seems inappropriate to me, but I typically use GLMs for binary and count data and want to make sure I am not missing some aspects of the use of contingency tables and chi squared tests. Below I will describe the general situation and outline a particular example.

My specific questions are

• Is my understanding/characterization of contingency table analysis correct?
• Is there a way the data I am describing could be analyzed correctly with a contingency table?
• How can I refine my explanations to best convey to students and collaborators the correct use of these methods?

Contingency table analysis of "counts" I think the problem might originate because when describing contingency table analyses, the term "counts" is frequently used to refer to the elements of each cell in a table. For example, the online Handbook of Biological Statistics describes a chi square test as appropriate "when you have 1 nominal variable with 2 or more values ..." and "You compare the observed counts of observations in each category with the expected counts." That is, when you have a nominal response variable (ie red, yellow, orange flowers) and you have counts of the number of times independent observations or trials have taken on each of those values.

In my experience, I think researchers can forget the "when you have one nominal variable" part and see only the word "counts"; that is, they use a chi-square or other methods (Fisher's exact test, G-test) on count data when they should use a linear model approach (t-test, regression, etc).

MADE UP EXAMPLE Say I go into a field and select a random point. Within 10 meters of that point there are 90 flowers. I go into a nearby forest, select a random point, and there are 30 flowers within 10 meters of the point. A chi square test in R

chisq.test(c(90,30))

Gives me a tiny p value of 4*10^-8. However, as I understand it, this test is inappropriate and there is no valid way to analyze these data. I have "counts" as the definition of a chi square test says I need, but they are not counts of the number of times a nominal response variable occurs; they are unbounded numeric data. To carry out a valid analysis, I would need replicated samples from each habitat. Is this correct?

REAL EXAMPLE In this paper on tropical birds, the author's used nets to capture birds at 2 different elevations. They want to know whether birds are more abundant at one elevation or the other. At each elevation they had 1 sampling location (no replication at a given elevation). At one elevation they caught 128 birds and at the other they caught 141 (Their Table 1, bottom row), and they want to test for a difference using a chi-squared test (Their Table 2, bottom row)). I believe if they had replicated sites they would've done a t-test, but since they only have one sampling location at each elevation they can't calculate a mean capture rate and have erroneously identified a chi-squared test -- perhaps because it works on "counts" -- as an alternative test.

EXPECTED COUNTS BASED ON SAMPLING EFFORT The previous example is not exactly how the authors carried out their analysis. In addition to the raw counts at each elevation(128 vs 141) they used information on the amount of sampling effort to formulate their expected counts. The amount of time the researchers spent catching birds was different between elevations; at one site they spent 287 hours and the other 157 (these are referred to as "net hours" - number of hours*number of nets). They state "comparisons were based on actual numbers (not rates) with expected values based on number of net hours." That is, they want to account for different amounts of sampling effort: 128 birds/287 net hours at one elevation vs. 141 birds/157 net hours at the other. They unfortunately to not report their test statistic nor exact p value, just p < 0.001, but I believe the R code pasted below replicates their workflow, where the expected number of birds at a site (E.i) was calculated as

E.i = (total birds caught during study)/(total hours of effort)*(hours of effort at site i)

I may very well be missing some aspect of how chi squared goodness of fit tests are used, but this does not seem to be correct. I believe the authors misunderstand the purpose of formulating "expected counts," again being mislead by the word "counts." As I understand it, the things being counted should be able to take on one discrete value or another, and the "expected count" is the number of times, given a number of trials or events, that one value should occur instead of the other. The expectation can be that a certain ratio occurs that is different from 50:50, say a sex ratio of 75:25. I believe this is where the authors think their weighting by sampling effort comes in, resulting in more captures where more effort was expended.

Calculations in R

#observations
obs1 <- 128; obs2 <- 141   
obs.tot <- obs1+obs2    

#sampling effort ("net hours")           
nethours1 <- 287; nethours2 <- 157
nethours.tot <- nethours1+nethours2

#expected counts based on effort
expect1 <- obs.tot/nethours.tot*nethours1
expect2 <- obs.tot/nethours.tot*nethours2

#ch2 test statistics: 
ch2 <- sum(c(((expect1-obs1)^2)/expect1, ((expect2-obs2)^2)/expect2))

#p value
pchisq(ch2,1,lower.tail = FALSE)

Answer 1

It may help to consider that there can be different kinds of counts. For example, if I flip a coin $n$ times, I can count how many heads I get. This could be $0$, $n$, or any whole number in between. Such data are binomial (or possibly multinomial). The point is that they have an intrinsic upper bound.

It is also possible for a count not to have a meaningful upper bound, such as the number of times a bird flies by my window over my lunch hour. There can be any number of birds, and some might fly back and forth many times. So while common sense suggests the number isn't really likely to be $18,\!000,\!000$, it could be an arbitrarily large number in the sense that I really cannot get $n+1$ heads out of $n$ flips.

This distinction is important for thinking about whether the chi-squared test makes sense. You shouldn't be using the chi-squared test if the upper bound isn't clear and known. Consider how the chi-squared test works when applied to a contingency table: glossing over quite a bit of detail, it divides various counts by the sum of the counts in the table to compute proportions. The test assumes the data are binomial / multinomial and that the table sum was the largest possible value. Mathematically, it is trivially true that the counts in the table will not be greater than the table sum, but those assumptions are nonetheless plainly incorrect when the upper limit is not bounded. When there is no upper bound, as in your examples, you aren't working with proportions of a total, but rates of some kind (e.g., flowers per $m^2$, etc.). At any rate, I agree with you that the chi-squared test is inappropriate for both your made-up flower example, and your real-world bird example.

On the other hand, it isn't quite true that such data cannot be analyzed. The most basic distribution for counts that are not bounded from above is the Poisson distribution. This often serves as a default distribution for such counts. A feature of the Poisson distribution is that the variance is equal to the mean. Thus, if you are willing to assume that the data would be distributed as Poisson in the population, you don't need $\ge 2$ counts in each condition, you could have only one and can still conduct a test (cf., my answer here: Two rate chi squared tests infection rate).

poisson.test(c(90,30))
#   Comparison of Poisson rates
# 
# data:  c(90, 30) time base: 1
# count1 = 90, expected count1 = 60, p-value = 3.773e-08
# alternative hypothesis: true rate ratio is not equal to 1
# 95 percent confidence interval:
#  1.964990 4.699155
# sample estimates:
# rate ratio 
#          3 

You can further add an offset to account for sampling effort. (An implication here is that you should not use a linear model or t-test.) However, this is flatly implausible. Count data are almost always overdispersed in practice, which means the p-value from this test is too low. So one way to use such a test is to put a plausible lower bound on the p-value.