I have several hundred years of church baptisms that will be searched by people wanting to find the baptisms of their ancestors. I want to call attention to periods in the records in which the number of baptisms is particulary sparse when compared to other periods, because sparse periods may indicate poor record keeping rather than an actual decrease in the number of baptisms. The baptisms were performed fairly often (every few days for busy churches), but not absolutely regularly, so the baptism dates are distributed somewhat randomly, and any number of baptisms could be performed on any given day.

One simple approach is to simply check for gaps of a specified length between baptism dates, but this would miss sparse periods in which baptisms were performed but much less often than usual. Perhaps the least biased approach is to graph the entire data set and let people decide for themselves what periods qualify as sparse, but this method has its own drawbacks associated with presenting the data to users, who would need to interact with the plot in order to scan the huge amount of data in it. I don't want to deal with that unless forced to do so.

What kinds of statistical methods are used in situations like this? Can anyone provide the name of an algorithm that has been designed for more or less this kind of problem, or describe an appropriate means of analysis? I'm willing to write code (Python) if necessary.


you could bucket your data into say 10 year intervals and then apply arima plus intervention detection culminating in the identification of "level shifts / step changes ". https://stats.stackexchange.com/search?tab=newest&q=user%3a3382%20level%20shifts%20step%20changes will give you some discussions and useful examples.

  • $\begingroup$ I hadn't heard of Arima before, so I did some reading. It seems to be for forecasting, rather than for identifying sparse regions in data sampling. I don't see how it would apply to my problem, and I could not find any examples of such an application in this forum or elsewhere. By "intervention detection" I am guessing that you mean the detection of a sparse region. That sounds right, but how to use Arima to do it is not clear. Can you point me to some specific examples perhaps? $\endgroup$ – kevinafra Nov 20 '18 at 4:19
  • $\begingroup$ Intervention Detection pdfs.semanticscholar.org/09c4/… deals with identifying deterministic changes in a time series. For example a time series (a collection of observations for a particular period) may have some autoregressive behavior but may also have pulse or step shifts. In your case this would be a downwards level shift e.g. 2.,2,2,2,2,1,1,1,1,2,2,2,2, The ARIMA portion of a model deals with endogenous effects while dummy indicators deal with exogenous effects which you sometimes can pre-specify but often need to empirically identify. $\endgroup$ – IrishStat Nov 20 '18 at 9:14
  • $\begingroup$ You might also see cran.r-project.org/web/packages/tsoutliers/index.html which can identify the deterministic structure BUT requires the pre-specification if the ARIMA component. Other available software simultaneously identifies both. $\endgroup$ – IrishStat Nov 20 '18 at 9:26
  • $\begingroup$ If you are satisfied with my answer , please accept it to close the question $\endgroup$ – IrishStat Nov 28 '18 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.