Detecting abnormal trends in timeseries data

I'm looking for an algorithm that can help identify abnormal trends in timeseries metrics. We offer several services which we monitor active usage against, for any given time of day (typically updated once a minute). Here's a example of one of our services: We do have systems in place like Nagios to detect major issues. But what I'm more interested in is gradual degradation that doesn't get noticed until ... well, it gets noticed. For example, if we zoom in to 12/6 we see: So what I'm specifically interested in here is the trend downward @17:30~. Likewise another good example would be: Here we can see the abnormalities on 12/1 and 12/4 during peak. Are there any formal methods of detecting this sort of trend (or rather, abnormaility to a historical trend)? Specifically interested in the trend here, not the raw numbers. So for example, these services have a hisotrical decreasing usage as time goes on - eventually, these numbers will drop from say 60k peaks to 15k peaks.

One approach would be to fit a regression model and examine points where the observed values depart from predicted values.

A common regression model for time series data with strong periodicity is ARIMA. Given high degree of granularity in your time variable, I suspect an ARIMA model would work very well.

These models have been implemented in most statistical packages. Here's an example using R:

install.packages('stats')
library(stats)

# display example data
plot(USAccDeaths)
points(USAccDeaths)

# fit arima model
fit <- arima(USAccDeaths, order = c(1,0,1), seasonal = list(order = c(1,0,1)))

# overlay fit
lines(USAccDeaths - fit$residuals, col='blue') # isolate observations with extreme residuals extreme_indices <- which(fit$residuals>quantile(fit$residuals, .9) | fit$residuals<quantile(fit\$residuals, .1))
extremes <- USAccDeaths
extremes[-extreme_indices] <- NA

# overlay isolated observations
points(extremes, col='red')

This produces the following graph, with unexpected observations highlighted in red: 