Zero Frequency and Small Sample Size

I have a situation where the skeletons from one of three archaeological sites do not display a bone condition called periostitis that is found at the other two.

The relevant statistics are:

Site 1: freq=.167, n=48
Site 2: freq=.000, n=5
Site 3: freq=.250, n=36.


Since it is unlikely that individuals from Site 2 would not have periostitis (for anthropological reasons), it seems reasonable that the small sample size (n=5) is to blame for the freq=0.

As a means of testing independence between the sites and frequency of periostitis, I performed a Fisher-Freeman-Halton test, which yielded p=.495; but, in accordance with the ASA statement on p-values, I'm trying to get away from null hypothesis significance testing (NHST) as the only support for my findings.

With other skeletal traits, I've showed that the overlap between the 95% CIs for various frequencies support the findings of the NHST, but the freq=0 reduces the value of this approach in this case. However, if I assume that Site 2 has a freq=.200 (which is the average frequency for all three sites), there is considerable overlap between the 95% CIs for the three sites, thereby supporting the NHST result.

This approach is reasonable since the 3 sites are close to each other geographically, and periostitis is a very common condition in prehistoric populations. Also, the binomial distribution shows a prob=.328 that samples of 5 would have no periostitis if the actual frequency is 1 in 5 (freq=.200).

Is this a legitimate analysis? Is projecting a frequency, based on logic but not supported by data, acceptable? Please give me your opinions.

The analysis of @bzki (+1) seems reasonable.

A brief Bayesian analysis might begin by noticing that you have seen 16 skeletons with periostitis out of 84 skeletons at nearby sites. So a reasonable posterior distribution of the rate $$\lambda$$ might be $$\mathsf{Gamma}(\text{shape}=16,\, \text{rate}=84),$$ which has mean $$0.190$$ and puts 95% of its probability in $$(0.109, 0.295).$$

In R:

qgamma(c(.025,.975), 16, 84)
[1] 0.1088736 0.2945264


So you should not be surprised to see no instances of periostitis at a site with only five skeletons.

• Thanx, BruceET...I appreciate the extra support for the belief that freq=0 is due to the small sample size and probably does not reflect the actual frequency of periostitis. – stevebyers2000 Nov 7 '19 at 15:43

I will only try to address the specific question about getting a confidence interval for site 2 where the estimated proportion is 0. A nice method to construct a confidence interval with good coverage properties in low-sample cases (like for site 2) is that proposed by Agresti and Coull (1998). Agresti and Caffo (2000) summarize that method here.

You can effectively add two pseudo observations to the successes and failures. I think this has some intuition similar to what you stated -- you are sort of doing a weighted average of the result you get (0) and 1/2, with the weighting dictated by the sample size.

Formally, following Agresti and Caffo (2000), the confidence interval would be given by: $$\tilde{p} \pm z_{0.025} \sqrt{\tilde{p}(1-\tilde{p})/\tilde{n}},$$ where $$\tilde{n} = n + 4$$ (and $$n = 5$$ for site 2), and $$\tilde{p} = (X + 2) / (n + 4)$$, where $$X$$ is the number of successes (0 for site 2).

Doing this you would get a confidence interval of $$(0, 0.49)$$ (I put a lower-limit at 0 for obvious reasons).

See this link for more detailed discussion.

Answering the more general question about what to do here -- producing a confidence interval for site 2 in the way I suggested is what I would do. There are probably some Bayesian approaches: suppose you observe site 1 and site 3, and these inform your prior expectation of the underlying proportion at site 2. But I think you have reason to expect your sites to have different underlying proportions (and actually this is a thing you are trying to assess), so I wouldn't actually go this route.

• Thank you for the input, bzki...I appreciate the suggestion, and will give it a try. – stevebyers2000 Nov 7 '19 at 15:47