Testing for a drop in bookings

Question

We're developing real-time alerts for fine-grained (every 5 minutes) time series bookings data, and I'm looking for the best approach to doing this. Idea is that if over the past 10–15 minutes (say) there's a big drop in bookings volume relative to some expectation, then I want to fire an alert. We've been using historical data to set the expectation.

I'm familiar with time series analysis and exponential smoothing. Holt–Winters came up as a possibility here, but even triple exponential smoothing appears to handle only one seasonality, and bookings data in my industry (travel) definitely has at least three driving seasonalities: daily, weekly and annual.

What we've been doing. One approach we've been using – and it seems to work OK in most cases – is to do a difference between two sample means hypothesis test. Say that we break bookings up into five-minute blocks. Then we build a size three sample over the past 15 minutes, a size 12 sample for the corresponding 15 minute spans over the past four weeks, and then test the difference in their means.

I am a concerned that the sample sizes (especially for the current bookings sample) are too small to be trustworthy. The test accounts for sample size, and anyway the variance within a sample is generally quite small, at least at volume, so maybe this is OK. But using an $n=3$ sample feels wrong. Besides this approach neglects the annual cycle, which I'd like to be able to capture if possible.

Extra background. A couple of considerations to add. Not sure about their relevance, but just in case:

1) These are global bookings (i.e. people all over the world book), so the bookings volumes are not simple sinusoids. They are roughly the sum of multiple bookings sinusoids corresponding to different regions (NA/LATAM, EU, APAC) in addition to those associated with the different seasonalities. I know we can analyze the individual regions to tease apart the sinusoids, and we want to do that. But also we want to know when the aggregate volume has dropped.

2) There is some sense in which bookings seem to have a binomial distribution. If we fix the user volume, then we can think of a booking as a "success" in a binomial trial. This comes up for example if you have a web server farm with many servers, and you look at the distribution of bookings counts across the servers within a short time window (e.g. 30 minutes). That distribution passes a normality test (I've been using Shapiro), and I assume it's because it's a binomial distribution given the underlying mechanism. Bookings over time aren't binomially distributed because the user volumes themselves fluctuate widely. But I still wonder if somehow we can use the large number of underlying binomial "conversion opportunities" as a way to drive an appropriate test.

My question is: what approach should we be using to test for bookings drops, given the above?

Update. Based on what Stephan says below (that this is really anomaly detection, not forecasting), I have another idea. What if I were to build a comparison sample based on historical 5-minute bookings counts (e.g. seven 5-minute blocks from the previous week, flanking and including the block corresponding to the current one, another seven from the week before that, and so on, maybe four weeks back, giving me an $n=28$ sample), and then use an interquartile range-based outlier test? Simple, and the detection bands are given by the boxplot whiskers. Is this at all plausible?

Answer 1

There are at least two more use cases for the kind of complex seasonality you have: electricity consumption and call center demand. Both have strong intra-day patterns and day of week seasonality, and at least electricity also has a big yearly effect (call centers probably less so). Both are typically analysed in 5- or 15-minute intervals. And both are active areas of research, so googling for "electicity forecasting" or "power forecasting" or "call center forecasting" may already be helpful. I particularly recommend this paper, which treats quite a few different methods.

Unfortunately, what you are interested in is not the point forecast per se, but anomaly or signal detection or monitoring. I don't recall anything like this in the forecasting literature, but there may be something in the signal processing literature. I suggest you use the keywords above and google for "signal detection in electricity time series" and similar.

If you don't find anything, I would recommend a simple and dirty approach: look at the Taylor article I linked above, fit one of the models and try to understand residual variance. It is homoskedastic, or does it vary over time, or with the underlying level? Calculate the fit and create some "reasonable" detection band around it, based on your understanding of the residual variance. Raise an alert if recent data are outside this band. You will need to experiment a little, anyway. Good luck!