# Testing for a drop in bookings

We're developing real-time alerts for fine-grained (every 5 minutes) time series bookings data, and I'm looking for the best approach to doing this. Idea is that if over the past 10–15 minutes (say) there's a big drop in bookings volume relative to some expectation, then I want to fire an alert. We've been using historical data to set the expectation.

I'm familiar with time series analysis and exponential smoothing. Holt–Winters came up as a possibility here, but even triple exponential smoothing appears to handle only one seasonality, and bookings data in my industry (travel) definitely has at least three driving seasonalities: daily, weekly and annual.

What we've been doing. One approach we've been using – and it seems to work OK in most cases – is to do a difference between two sample means hypothesis test. Say that we break bookings up into five-minute blocks. Then we build a size three sample over the past 15 minutes, a size 12 sample for the corresponding 15 minute spans over the past four weeks, and then test the difference in their means.

I am a concerned that the sample sizes (especially for the current bookings sample) are too small to be trustworthy. The test accounts for sample size, and anyway the variance within a sample is generally quite small, at least at volume, so maybe this is OK. But using an $n=3$ sample feels wrong. Besides this approach neglects the annual cycle, which I'd like to be able to capture if possible.

Extra background. A couple of considerations to add. Not sure about their relevance, but just in case:

1) These are global bookings (i.e. people all over the world book), so the bookings volumes are not simple sinusoids. They are roughly the sum of multiple bookings sinusoids corresponding to different regions (NA/LATAM, EU, APAC) in addition to those associated with the different seasonalities. I know we can analyze the individual regions to tease apart the sinusoids, and we want to do that. But also we want to know when the aggregate volume has dropped.

2) There is some sense in which bookings seem to have a binomial distribution. If we fix the user volume, then we can think of a booking as a "success" in a binomial trial. This comes up for example if you have a web server farm with many servers, and you look at the distribution of bookings counts across the servers within a short time window (e.g. 30 minutes). That distribution passes a normality test (I've been using Shapiro), and I assume it's because it's a binomial distribution given the underlying mechanism. Bookings over time aren't binomially distributed because the user volumes themselves fluctuate widely. But I still wonder if somehow we can use the large number of underlying binomial "conversion opportunities" as a way to drive an appropriate test.

My question is: what approach should we be using to test for bookings drops, given the above?

Update. Based on what Stephan says below (that this is really anomaly detection, not forecasting), I have another idea. What if I were to build a comparison sample based on historical 5-minute bookings counts (e.g. seven 5-minute blocks from the previous week, flanking and including the block corresponding to the current one, another seven from the week before that, and so on, maybe four weeks back, giving me an $n=28$ sample), and then use an interquartile range-based outlier test? Simple, and the detection bands are given by the boxplot whiskers. Is this at all plausible?

• This sounds Poisson-y to me, which would make sense. Try normalizing residuals by fits: $(y-\hat{y})/\hat{y}$ (dealing with small $\hat{y}$ separately) and examining the variance of these transformed residuals. This essentially models residual variance as a linear multiple of the underlying mean. – Stephan Kolassa Apr 3 '14 at 7:27
• I kind of hope that the normalized residual variance is homoskedastic and does not exhibit any patterns any more. If so, let $s^2$ be the mean of the squared normalized residuals. You can now forecast bookings for the next couple of time buckets and use $k\times\hat{y}\times s$ as the width of the detection band, where $\hat{y}\times s$ is a "Poisson-y" level-dependent standard deviation, and $k$ is a parameter you could tune to get the desired amount of alerts. – Stephan Kolassa Apr 3 '14 at 7:39