3
$\begingroup$

I am trying to find Outliers in a contextual time series data using ARIMA model.

My data contains the Hourly Average Speed and Volume of vehicle traffic for two months. So, day 1 (say Monday) would have 24 observations and so on. There is seasonality in my data : Volume is really high at 08:00 and again high at 17:00 on the weekdays.

So, how I planned to do it was, take a Monday that has no outliers, use that as an input the the ARIMA model to predict the next 24 hours. And compare this prediction with the other Mondays that I have to find outliers in those Mondays that I compare. Similarly take a Tuesday that has no outliers, input that to ARIMA to predict the next 24 hours and compare that prediction to the Tuesdays I have. Do the same for Wednesdays to Sundays. Also, take holidays (since they have different vehicle count to that of working days) and compare the forecast of a holiday with the other holidays I have. [I am not sure if it makes sense but that is what I have in mind].

Will my idea work or are there some other suggestions?

And, how would I use ARIMA for hourly data?

Data in CSV format [Month - 1 and 2]

There are about 2 x 24 x 30 values in total 
$\endgroup$
  • $\begingroup$ You might want to look at demand-planning.com/2010/03/18/… for some hints. $\endgroup$ – IrishStat Nov 2 '17 at 18:30
  • $\begingroup$ It appears that you have 6 distinct time series (1,..6) and two characteristics (measurements) . Which one are you trying to predict/analyze/cleanse ? Post an actual data file containing 60x24 values for 1 time series and 1 characteristic where 60 is my guess as to the # of days you have since you reported two months thus 1440 values in a single column. $\endgroup$ – IrishStat Nov 3 '17 at 17:25
  • $\begingroup$ please on;y post the 1440 values (for 1 characteristic/measurement) showing date/hour .... post as an attachment not text $\endgroup$ – IrishStat Nov 3 '17 at 18:00
  • $\begingroup$ @IrishStat Thanks, I have added an attachment. The data is hourly, and I have just taken 1 time series (for one of the lanes). Since, the data is hourly and I have only posted one months data, it is about 24 x 30 values. $\endgroup$ – RPT Nov 3 '17 at 18:57
  • $\begingroup$ @IrishStat I have added the data for the second month as well, thanks $\endgroup$ – RPT Nov 3 '17 at 19:07
1
$\begingroup$

I took your 61 consecutive days of data (24 hourly readings per day) enter image description here. The 1464 values were not analyzed in one model because there were essentially 24 sets of 61 historical values piggy-backed together. ACF/PACF analysis makes little sense as for example. the 25th value has little or nothing to do with the 24th value etc. thus temporal autocorrelation is non-informative. The fact that the 24th value might be related to the 48th value is much more interesting or even the 192nd value (1 week later).

I analyzed it using AUTOBOX a piece of software that I have helped develop using a 30 day forecast horizon. The documentation for the approach can be found in the User Guide available from the AFS website. I will try and give you you a general overview here and to tie it in to your intentions to want to use daily totals (and forecasts of same) to guide hourly forecasts. Mixed seasonal problems like yours can easily be mis-modelled thus over many years of forecasting 15 minute intervals, hourly intervals etc data we have formed an integrated solution. As an overview we use daily aggregates as a predictor to hourly values thus in this case some 25 models were developed. Ultimately the 24 hourly forecasts are reconciled with the daily forecasts yielding a final solution to this "thorny problem".

The data is analyzed in a parent-to-child approach where a model is initially developed for the daily totals [enter image description here] and here enter image description hereincorporating memory,daily effects and anomalies, level shifts , local time trends etc.. This "parent model" leads to forecasts and confidence limits based upon possible daily totals. Note that day 3 is a Saturday and day 4 is a Sunday showing significantly lower values for Volume ( shown as X1 and X2 in the equation)

The next step is to identify 24 causal models using the parent i.e. the daily total and it;s forecasts as a possible predictor using memory as needed , level shifts as needed while identifying and remedying possible anomalies/level shifts/local time trends. As an example of this let me show you the graphical output for hour 12 enter image description here enter image description here . Now we have 24 sets of forecasts for the children for the next 30 days and a set of forecasts for the parent for the next 308 days. We reconcile these two to obtain the final forecasts for 24 hours for the 30 day forecast.

The reconciliation can be done in a parent-to-child or a child-to-parent manner. Clearly 61 days of data insufficient to capture , weekly effects, holiday effects , specific days of the month effects , long-weekend effects , week-in-month effects, monthly effects etc.. but if you had a longer series you can get the picture as to what might be possible. The detailed output can be made available to you or any interested party by contacting me as it is to voluminous to post. With this analysis one might want to creatively program the approach with free R tools but there are a lot of pitfalls awaiting such enterprise. Hope this helps your research into what I think is a very important statistical problem regarding detecting exceptional events and accounting for their effect in the forecast horizon.

I have made similar responses in the past .. here where hour is a sub group Robust time-series regression for outlier detection and here for product/class sub-group analyses Forecasting Amazon or Netflix demand

It is important to note that outliers reflect the impact of exceptional exogenous activity GIVEN other factors thus the importance of correctly encoding daily effects i.e. the kind of day . Longer time series would reveal holiday effects and other "causals" .

$\endgroup$
  • $\begingroup$ thank you very much for your detailed answer. This is what I was hoping for. Really appreciate it. Also, can I have access to the detailed report as well, please? Thank you $\endgroup$ – RPT Nov 4 '17 at 20:00
2
$\begingroup$

Don't use ARIMA. You have (namely, day-over-day and week-over-week), and ARIMA cannot handle this. models can. I'd recommend you look at previous threads in these two tags.

Similar multiply-seasonal patterns occur in electricity demand and in call center demand data. Looking at modeling and forecasting literature for these two use cases may be helpful.

You can fit a TBATS model and then look at residuals to detect outliers.

$\endgroup$
  • $\begingroup$ Many thanks for your reply. But can I ask why my data would have multiple-seasonalities when I am just passing one day(24 hour) worth of data to the model to predict the next 24 hours? Is it not just an hourly seasonality? I am sorry but I am quite new to this, if my question does not make much sense. Also, do you recommend any reading on time-series forecasting? Thanks very much. $\endgroup$ – RPT Nov 3 '17 at 10:39
  • 1
    $\begingroup$ From your question, it sounded as if you had way more than just one day's worth of data, don't you? In this case, throwing everything but one day's worth away is not a good idea. Better to use all your data, with an appropriate model. As a textbook, I recommend Forecasting: Principles and Practice by Athanasopoulos and Hyndman, which is freely available online. Section 11.1 covers complex seasonality. $\endgroup$ – Stephan Kolassa Nov 3 '17 at 10:43
  • $\begingroup$ Sorry if my question sounded a bit ambiguous. My aim is to find outliers in daily vehicle traffic data. So, what I thought of doing was, take one monday's data that has no outliers, forecast it using ARIMA (since it has only hourly seasonality), compare the prediction with the other mondays in the data to find outliers in them. Then get one tuesday's data with no outliers and repeat the same. Does it seem okay to do this? Or, if you suggest using all the data, can you say why? Thank you very much :) $\endgroup$ – RPT Nov 3 '17 at 10:49
  • $\begingroup$ @R.p.T, perhaps you could phrase it in more detail / more clearly by editing your post. $\endgroup$ – Richard Hardy Nov 3 '17 at 11:47
  • $\begingroup$ @RichardHardy, I have edited my question and tried to make it more clear. Thanks $\endgroup$ – RPT Nov 3 '17 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.