1
$\begingroup$

I am working on a anomaly detection for a batches of daily time series (non-hierarchical) that exhibit both yearly and weekly seasonality. I tested a few algorithms and it appears that tbats() from the forecast package works best. I find it captures weekly seasonality and the trend captures the yearly seasonality via levels.

I have to split my data into the most recent 90 days (this data is not "finished" and is still having revenue trickle in). As such, I go back 365 days to get a historical trends. This denotes the "historic" and the "recent data sets" data sets. I will define recent data points that are way below the tbats value as outliers.

#Load Libraries
library(data.table) #Data cleaning Library
library(lubridate)  #Date cleaning Library
library(dplyr)      #data manipulation Library
library(zoo)        #time series indexing Library
library(stats)      #Statistics Library
library(timeSeries) #Time Series Library
library(forecast)   #Forecasting Library
library(purrr)      #data cleaning library

#orders dates oldest to newest for each department
orderedhistoric <- arrange(historicdata,Department,ServiceDate)
orderedrecent  <- arrange(recentdata,Department,ServiceDate)

This just arranges it so that the data is ordered for the ts() function

#splits into a list of department data
historicsplit <- split(orderedhistoric, orderedhistoric[,2], drop = FALSE)
recentsplit  <- split(orderedrecent, orderedrecent[,2], drop = FALSE)

This splits it by product (my 2nd column)

#Creates timeseries elements for every department
historictimeseries <- lapply(historicsplit,function(x) ts(x[3],frequency=7))
recenttimeseries <- lapply(recentsplit,function(x) ts(x[3],frequency=7))

This makes it the daily timeseries data with weekly seasonality

#finds optimal TBATS from historic data
TBATS <- lapply(historictimeseries, function(x) tbats(x))

I am running into trouble on fitting these tbats parameters to the corresponding recent data. I basically want to do a standard cross validation on this, despite it being time series data. The recent data hasn't had all the revenue posted (in fact this is what it's trying to find), so I definitely don't want it to influence the tbats parameters.

Alternatively, I would take other suggestions on how to do this. Again, I am using TBATs as opposed to auto.Arima due to complex seasonality.

$\endgroup$
  • $\begingroup$ Shoot, this should be at regular stack exchange, since it's more of a coding question. Apologies $\endgroup$ – Ben Feb 14 at 19:13
0
$\begingroup$

There are packages and functions designed to do anomaly detection. It sounds like you’re creating a forecast and then subjectively identifying outliers. Lately I’ve been using tsoutliers because I like its simplicity and plots. Within the packages you’re already using there are options too.

Also, There is an msts() function where you can select multiple seasonal frequencies. Try that instead of ts().

$\endgroup$
  • $\begingroup$ I'll try TS outliers, but ARIMA was not fitting my data well. It focuses too much this date vs last date. I was using 2 standard deviation from the mean residual to find outliers, and correctly finding holidays and ice storms, when we would anticipate less sales. I think STL or something might be worth a shot too. $\endgroup$ – Ben Feb 15 at 12:21
  • $\begingroup$ I haven’t personally tried it so I’ll take your word for it. Have you used msts() when using ARIMA though? Perhaps explicitly making both seasonal frequencies part of the time series object will improve ARIMA $\endgroup$ – Chris Umphlett Feb 15 at 12:48
  • $\begingroup$ I'll try uses msts using ARIMA, but I was adding a fourier term to the ARIMA and that was not doing the greatest job. I have a couple true outliers (Christimas day etc.) that are being caught with tbats but not with auto.arima (this might work perfectly fine on other data sets). I find that the yearly seasonality is being caught with the "trend" components of tbats. If I were to venture a guess, ARIMA isn't doing this quite as effectively. I will give msts a try. though since it's a year of data the yearly seasonality might be better caught as a trend. $\endgroup$ – Ben Feb 15 at 13:55
  • $\begingroup$ While it may be working the trend should not be modeling seasonality but rather the change in the level. If your series has a trend then the trend component may get confused between what is a seasonal change and what is a true shifting of the level. Regardless of whether you use an ARIMA based approach you should aim to model both seasonal frequencies appropriately. $\endgroup$ – Chris Umphlett Feb 15 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.