I am new to both ARIMA technique and R. Your suggestions would be very much appreciated.

I am dealing with hourly data with strong seasonality and have used auto.arima to select a model:

Fit<-auto.arima(H_ts, seasonal=TRUE ,approximation=FALSE)

This is the model it returns: ARIMA(5,1,4)(1,0,0)[24]


         ar1     ar2     ar3      ar4      ar5      ma1      ma2      ma3     ma4    sar1
      0.6361  0.4046  0.5212  -0.7154  -0.0334  -0.5638  -0.4464  -0.5704  0.6325  0.0917
s.e.  0.0977  0.1390  0.1109   0.0968   0.0181   0.0957   0.1282   0.1152  0.0821  0.0114

sigma^2 estimated as 14.24:  log likelihood=-12188.98
AIC=24399.96   AICc=24400.02   BIC=24470.34

However, the residual ACF and PACF look quite suspicious: enter image description here

Any idea what I can do to 'fix' this?

  • $\begingroup$ Can you get nicer residuals if you specify the frequency of your time series to be 23 or 22 in place of 24? Nothing much behind this, just looking at the spikes around these lags... $\endgroup$ – Richard Hardy Nov 28 '16 at 20:03
  • $\begingroup$ The residual acf clearly indicates that it is not white noise. There seems to be a residual seasonal component. As Richard Hardy suggests it may be that the seasonal period is different from 24 hours. $\endgroup$ – Michael Chernick Nov 28 '16 at 20:58
  • $\begingroup$ spikes around the correct lag are quite freguent if there is also a need for short term ar structure. Hourly data often has fixed daily patterns and often weekly/monthly patterns and of course holiday effects etc. . The model you formed is probably quite incorrect based upon a ton of experiences with hourly data. I should also say that the s.e. of the acf is 1/sqrt(n) which for large N generates false (tight) limits. Remember that 1/sqrt(n) is very naive. If you post your data in 24 csv files and report the start date/country or origin/kind of data I will take a look at it. $\endgroup$ – IrishStat Nov 28 '16 at 21:17
  • $\begingroup$ @IrishStat I would love to take up on your offer, but how do I share my data? (I am new to this site as well). $\endgroup$ – C.Woo Nov 28 '16 at 23:01
  • $\begingroup$ I don't really know to how to post a csv file (with 24 data columns) so I can't be of much help. If you want you can email it to me at my contact info . Perhaps someone else here can help with this . $\endgroup$ – IrishStat Nov 28 '16 at 23:37

Thanks for sharing your data (available from me) detailing 24 hours of data for 884 consecutive days (start date: 5/1/14) representing emergency room visits to a hospital. Your stated task was to simply predict the next 8 hours. We have routinely developed forecasts like this for manpower planning purposes. I believe by using ARIMA methods you are asking the wrong question . An ARIMA model uses what occurred in recent hours and recent days via seasonality of 24 and 7 to project forward based upon these previous values. Essentially the answer is to the question "How do I use previous values to make a weighted average forecast"

The truth of the matter is that you should make a forecast for the next 8 hours taking into account 1) the hour effect , 2) the day-of-the-week effect and changes in the day-of-the-week effect , 3) the month-of-the-year effect , 4) holiday effects , 5) possible long-weekend effects , 6) overall global trends/level shifts affecting the hospital while also adjusting for any additional memory effects from previous days all while identifying/ignoring unusual values as not being representative of process. In this way you are identifying possibly significant assignable cause to a number of possibly usefuly deterministic variables.

Although you only wanted the next 8 hours for the next day , I used 24 hours for the next 365 days to help illustrate the guidance available from this approach . Please review Time series model of intraday data on weekdays and weekends and it’s threads as it will help you understand the general direction of mixed frequency modeling … see my comment and referenced url’s

I took your daily total series and automatically developed a model (presented at he end of this response ) which generated forecasts to guide the hourly predictions. This is the plot of the Actual/Fit and Forecasts for the Daily Totals.enter image description here . The model is interesting as it identifies the major level shift in the series , the downside predictions for Saturday and Sunday , and the monthly effects suggesting changing emergency through the year. Here is the model in two parts ..

Now 24 individual model were developed using the GROUP(DAILY) TOTAL as a possible predictor. Like children some follow the parent more closely and some don't.

I present here the Actual/Fit/Forecast for hour 1 (12:00 pm to 1:00 am ).

enter image description here . The model for Hour 1 is here enter image description here and Hour 8 is here enter image description here . It is interesting ( at least to me ) how different they are in terms of unusual values and varying days-of-the-week effects sort of suggesting that Hour1 is easily predicted while Hour8 is less easily predicted due to much higher volume levels, daily effects and an increased # of random/unpredictable arrivals.

The ACTUAL/FIT/FORECAST respectively (note the scaling is different ... don't let your eye be fooled ! )

enter image description here and enter image description here

After reconciling the children to the parent ( an option ) here are the forecasts for the next 8 hours (13 hours shown for day 1 in the future)

enter image description here...

enter image description here and here enter image description here . Recall all models are wrong and some are even worse. The important thing is that existent domain knowledge ( auxiliary/helping X's ) should be used wherever possible to improve comprehensive understanding e.g. population trends , competition , price etc .


enter image description here

  • $\begingroup$ Thanks for your timely response. I have some follow-up questions: 1. Just to clarify: are you suggesting that instead of using arima, I should build a model for each hour taking into account things like day of the week, hour of the day, holiday, time of the year, etc.? 2. What is the GROUP(DAILY)TOTAL that you used? Is it available in R? (that’s the only software available to me) Thanks! $\endgroup$ – C.Woo Nov 29 '16 at 20:29
  • $\begingroup$ The Groupt variable is the total of all 24 hours providing a general guidance variable. Specifically you would 24 equations .. one for each hour where you specified as possible predictor variables; Groupt and , Day-of-the-week allowing for anomalies and possible ARIMA structure to pick up any needed short-term memory. You would never use time of the year and you don't need holidays as that effect has been built into Groupt $\endgroup$ – IrishStat Nov 29 '16 at 20:39
  • $\begingroup$ I added the acf/pacf for HOUR1's model to show how there is no remaining structure i.e. the errors appear to be wn $\endgroup$ – IrishStat Nov 29 '16 at 20:45
  • $\begingroup$ I have certainly learned a lot from this discussion. Your model looks quite nicely done. What software do you use? If you don't mind sharing your code that'd be great. Thanks again. $\endgroup$ – C.Woo Nov 29 '16 at 22:27
  • $\begingroup$ demand-planning.com/2010/03/18/… discusses the mixed frequency approach of AUTOBOX autobox.com/cms which I helped develop. There is an R version which you can freely download and play with. $\endgroup$ – IrishStat Nov 29 '16 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.