Test for spacing of regularly occurring events measured by several unreliable observers Let me see if I can explain this problem in an intuitive way.
Suppose I have a lighthouse in the distance, and it's either flashing every 1 second, or every 5 seconds. I want to determine which it is.
I have a group of people that are going to independently (i.e., not at the same time) observe the lighthouse for 60 seconds each. They will write down times in seconds relative to the beginning of their observation when they see the light flash. (So, if they see the light at 8 seconds and 21 seconds, they'd write down 8 and 21.)
If my observers were perfect, this would be easy. But unfortunately, they are not. They're highly unreliable and most of them have a less than 10% chance of seeing any given flash. (Consider consecutive flashes independent. Seeing flash N does not affect the observer's chance of seeing flash N + 1.) And each observer has a different miss rate. Some might see the light 10% of the time, others might see it 1% of the time, etc.
The observers are also not perfect about measuring time. Sometimes, they'll be off by a second or two, though 95% of their measurements are off by 1 second or less. This property is the same for all observers - their time measurement abilities can be considered equal for the purposes of this problem.
Given this data, how can I design a statistical test to determine whether the lighthouse is flashing every 1 second or every 5 seconds?
My first intuition was to measure the gaps between sightings for each observer. So, if an observer saw the lighthouse at 6, 11, 26, and 51 seconds, that would produce gaps of 5, 15, and 25 seconds. I could then take these gaps, divide them by 5, and get the remainder. In this case, the remainders would be 0, 0, 0. I could then plug this into a chi-square test to see if the gaps-mod-5 are evenly distributed, or if they're concentrated around 0.
But there's two problems with this. First, you can't assume that the gaps-mod-5 should be evenly distributed between 0 and 4 in the lighthouse-flashing-every-second case, because long gaps should be relatively less likely than short gaps (since a long gap implies missing more flashes than a short gap). And the expected distribution of the gaps is dependent on the miss rate, which is different for each observer... So, when you assume the gaps-mod-5 should be evenly distributed between 0 and 4, the chi-square ends up rejecting a lot because you're seeing 0 less than 20% of the time, rather than more. But that's expected.
The second problem with using the remainders is that you lose some of your power. A gap of 45 seconds is good evidence for the lighthouse flashing every 5 seconds not just because it's a multiple of 5, but also because it implies that you would have missed 44 consecutive flashes in the 1-second case. If the observer in question sees the light 10% of the time, missing 44 consecutive flashes is highly unlikely. Of course, the miss rate is also unique to each observer, and it's an unknown value.
The other intuition I had was that you could estimate the miss rate for each observer for the 1-second case by taking the number of sightings and dividing by 60. If observer X saw the lighthouse twice, that's an implied miss rate of 58 / 60 = 97% if the lighthouse is flashing every second. You could then calculate for that particular observer an expected distribution of observed gaps, where frequency of each gap length should just be a function of the estimated miss rate. Then you could sum your expected and observed values across observers. Then you could run your chi-square. Unfortunately, I don't know how using estimates of your expected values would affect your chi-square results...
Does anybody have suggestions for another test that could be used to elegantly handle this problem? It has relations to the poisson distribution as well, but I'm not sure how to design a neat test to fit there either.
 A: I wrote up the below before noticing the condition that the observers aren't perfect timekeepers. This answer assumes they do keep time perfectly, but I'm posting it anyway as a similar approach might still go somewhere.
Let $R_k$ be the hypothesis that there are $k$ flashes per minute. Suppose the $i$-th observer observes $\textbf{n}_i$ flashes, at times $\textbf{X}_i$. Here $\textbf{n}$ is a vector and $\textbf{X}$ is a two-dimensional jagged array. Without loss of generality assume that flashes happen exactly on the second, so that every observation is recorded as happening at an integer number of seconds between $0$ and $59$ inclusive.
We are interested in the likelihood ratio:
$$
L = \frac{\mathbb{P}(\textbf{n},\textbf{X}|R_5)}{\mathbb{P}(\textbf{n},\textbf{X}|R_1)}
= \frac{\mathbb{P}(\textbf{n}|R_5)}{\mathbb{P}(\textbf{n}|R_1)}\prod_i\frac{\mathbb{P}(\textbf{X}_i|\textbf{n}_i,R_5)}{\mathbb{P}(\textbf{X}_i|\textbf{n}_i,R_1)}
$$
For the first factor: $\mathbb{P}(\textbf{n}|R_k)$ is the probability that for each $i$, the $i$-th observer sees $\textbf{n}_i$ flashes out of a possible total of $\frac{60}{k}$ flashes. This is something you would have to put a number on based on your prior knowledge of the observers.
The other factors can be calculated exactly:
$$
\frac{\mathbb{P}(\textbf{X}_i|\textbf{n}_i,R_5)}{\mathbb{P}(\textbf{X}_i|\textbf{n}_i,R_1)} = \begin{cases}\begin{align}0\:\: &\text{if any two observations are not a multiple of 5s apart} \\
1\:\: &\text{if } \textbf{n}_i = 0\\
\frac{60 \choose \textbf{n}_i}{5 {12 \choose \textbf{n}_i}}\:\: &\text{otherwise}\\
\end{align}\end{cases}
$$
Justification for that final expression with factorials: if there is a flash every second, there are $60 \choose \textbf{n}_i$ possible ways to observe $\textbf{n}_i$ flashes; if there is a flash every 5 seconds, there are $5$ possible choices of equivalence class modulo $5$, and $12 \choose \textbf{n}_i$ possible ways to observe $\textbf{n}_i$ flashes at times belong to the chosen equivalence class.
Finally Bayes' Theorem gives us:
$$\mathbb{P}(R_5|\textbf{n},\textbf{X}) = \frac{L\mathbb{P}(R_5)}{L\mathbb{P}(R_5) + \mathbb{P}(R_1)}$$
$L$ will often be large (when all the observations are consistent with $R_5$ and some observers record multiple flashes) or exactly zero (when the observations are at all inconsistent with $R_5$ due to an observer recording two flashes that aren't separated by a multiple of five seconds).
