I have a bunch of count data associated with time intervals (potentially overlapping and of variable lengths), say

$(s_i, t_i, n_i, N_i)$

where $N_i$ is a count of the total number of events observed by a particular observer over a time interval $[s_i, t_i]$, and $n_i$ is the number of those events of some class of interest.

I'm interested in modelling how the proportion of interesting events varies over time, assuming that this proportion is independent of the observer. The total number of events observed in each interval may vary somewhat arbitrarily based on the observer, although if necessary we can safely assume that in aggregate over all observers it gives a reasonable picture of the overall event density.

The simplest model I could think of for this data would be to assume $n_i \sim Binomial\left[N_i, \frac{1}{t_i-s_i}\int_{s_i}^{t_i}\theta(w) dw\right]$

For some unknown smooth function $\theta(t)$ which lies in $[0,1]$ for all $t$, and look for non-parametric methods to estimate $\theta$.

A slightly better model could perhaps be to assume an inhomogeneous Poisson process for the overall events with rate $\lambda(t)$, and that the events of interest are a thinning of this process in a time-varying proportion $\theta(t)$ yielding another inhomog. Poisson process with rate $\lambda(t)\theta(t)$. We can further assume that both these processes are thinned further by each observer, independently to the "event of interest" thinning, in some unknown fixed proportion representing the proportion of events of both types which the observer sees. This proportion cancels out leaving:

$n_i \sim Binomial\left[N_i, \frac{\int_{s_i}^{t_i}\theta(w)\lambda(w) dw}{\int_{s_i}^{t_i} \lambda(w) dw}\right]$.

Here $\lambda(t)$ only matters up to a constant factor, so perhaps it would be better to model it as a density rather than a Poisson process rate? Anyway the simpler model falls out as a special case of this if you assume $\lambda(t)$ is constant/uniform.

I was wondering if anyone has any pointers on how to estimate the proportion function $\theta(t)$ in this kind of model?

A naive ad-hoc guess would be to compute a separate estimate for the overall rate and the rate of interest using kernel smoothing methods, then compute a ratio of these two estimates (perhaps adding on some small constants akin to what you'd get using a Beta prior to estimate a standalone binomial proportion e.g. $\frac{n+\alpha}{N+\alpha+\beta}$).

To bring out the heavy guns, I suppose one could put a logistic Gaussian Process prior on $\theta$ and estimate look for a posterior estimate. I imagine this would require some fairly hairy approximate Bayesian methods.

I was wondering there were any simpler frequentist estimators for this (based for example on kernel density estimators), and if anything is known about hyper-parameter selection, convergence rates, confidence intervals and so on.

If it helps to consider a simpler problem first, we could perhaps assume that we have point observations for the events (of interest and not).


  • $\begingroup$ To partially answer my question, it seems a good way of looking at this is to view this as an inverse problem, with a binomial noise distribution. $\endgroup$
    – Matt
    Apr 1, 2014 at 17:22
  • $\begingroup$ I am hoping the answer (or at least the right tools) lie in part in this paper: Inverse problems as statistics live-stat.pantheon.berkeley.edu/sites/default/files/… . Here the "blurring" operator is the process of computing the integral of the latent function $\theta(t)$ over a bunch of different observed time intervals (handily this is linear), and the noise arises from a Binomial draw rather than e.g. additive Gaussian noise. It seems inverse problems with Poisson noise are common e.g. when dealing with photon counts, so can hopefully cope with Binomial noise. $\endgroup$
    – Matt
    Apr 3, 2014 at 18:43


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.