1
$\begingroup$

I have a set of 3 years of daily data. I saw weekly and annual seasonality in the data so I used msts time series and tbats (from the forecast package in R) to fit the best fitted model.

The predicted values for weekdays are with 5% of the actual data but it has very off predictions for weekend. I did not expect that as I included daily seasonality in a week (different weekday and weekend patterns) in my time series which I though will consider the seasonality correctly. I wonder if anybody have any idea whats going wrong with my data.

I also used ts with single seasonality of frequency 7 and again used tbats to fit a model. Th new model has better predictions for weekend but worse predictions for weekday. I also tried auto.arima (also from the forecast package) but as I have a huge number of data points, arima was not able to find a good model.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ There's unlikely to be enough information here for anyone to identify what the problem might be. I'd start with a plot of your data, or at least part of it, as well as corresponding fitted values (that still won't be sufficient, but it may give some clues or enable people to suggest some things to try) $\endgroup$
    – Glen_b
    Commented Sep 22, 2014 at 22:48
  • $\begingroup$ Can you tell us more about the data? $\endgroup$
    – jenny
    Commented Sep 24, 2014 at 15:25
  • $\begingroup$ Well you didn't mention that until now (the shipments part). Can you more mathematically define how you handled the seasonality patterns to help us understand what might have gone wrong? $\endgroup$
    – jenny
    Commented Sep 24, 2014 at 15:37
  • $\begingroup$ I used msts command in r by including seasonality of 7 for weekly seasonality and 365.25 for annually seasonality. Is that what you meant? $\endgroup$
    – user12
    Commented Sep 24, 2014 at 15:45
  • $\begingroup$ why don't you provide the data via dropbox $\endgroup$
    – IrishStat
    Commented Sep 24, 2014 at 16:21

2 Answers 2

Reset to default
3
$\begingroup$

Some exploratory data analysis: In summarizing your data at the monthly level, I noticed you have couple of issues going on in your data.

  1. In the test period data which begins at January 2010 there is a significant "bending upwards" of trend.
  2. In addition I also see there is a high degree of variability in seasonality, so you might want to apply an appropriate transformation using box-cox transformation.

Coming back to the first part of the issue, before using an extrapolation method such as ARIMA or exponential smoothing or fixated on data mining/dredging, I would recommend to find out "why" the trend changed direction and went upwards. Unless you know the "why" part of this trend change, no matter what method you use, it is going to be impossible to improve accuracy or build better forecast. As noted in answer to your earlier question, the only way to build good forecasts for your problem is to systematically adjust the forecast from an extrapolative method using a well structured judgement or use an analogous time series top forecast time series. You could also use an ensemble of judgemental forecasting and extrapolative methods. Using univariate extrapolative method alone on a non-experimental data such as yours bound to produce poor predictions. It is important to do some initial data analysis before fixating on methods and techniques, it might go a long way in improving predictions. Also, using a low frequency data such as rolling the daily data to monthly data and doing data visualization also is very helpful to better understand the data and ultimately better predictions.

Hopefully this analysis provided you some directions for future research.

I have to agree with Whuber, that only way to assess predictive performance of an extrapolation method is using hold out set. Simplest is the single origin forecast,If you have enough resources then I would do rolling forecast testing or cross-validation or jackknifing such as the one suggested by @Irishstat. See the link below for some nice blog post by Rob Hyndman on how to do this for time series data.

http://robjhyndman.com/hyndsight/tscvexample/

http://robjhyndman.com/hyndsight/rolling-forecasts/

I'm also curious to know why you would need a daily forecast for 1 year. In practice based on my own experience, you would use low frequency data (such as monthly/weekly) to do a full year forecast for planning purpose and use high frequency data such a hourly/daily to do a very very short term forecast. When I mean very very short term, update and revise forecast every month. As an example, If I had your data (assuming you want to forecast for 2010), I would predict full year 2010 data using monthly data, and just produce daily forecast for January 2010. By this use you could get the bests of both worlds by have low frequency (less noise) and high frequency (daily data) forecasts. Once you have Jan 2010 data, you could produce daily forecast for February and keep producing and updating the daily forecast. You could repeat this for every month as the new data arrives. In addition you could also reconcile the monthly and daily forecast.

enter image description here

Improve this answer
$\endgroup$
4
  • $\begingroup$ Box-Cox transformations should be based upon the linkage between the error variance and the expected value NOT the variance of the original series.. Also your remarks about forecasting out 366 values and measuring accuracy for 366 values echoes my comments above. It is not naive to do this but simply inappropriate,in my opinion. $\endgroup$
    – IrishStat
    Commented Sep 29, 2014 at 22:19
  • $\begingroup$ As noted in my post, single origin forecast evaluation is one of the several options you could also do cross validation or rolling forecast evaluation. Out of curiosity, does Autobox provides multiple origin forecast evaluation ? $\endgroup$
    – forecaster
    Commented Sep 29, 2014 at 22:54
  • $\begingroup$ AUTOBOX does not do this automatically but it does allow the user the repetitive usage of a model from multiple origins. $\endgroup$
    – IrishStat
    Commented Sep 30, 2014 at 2:09
  • $\begingroup$ @forecaster thank you for your discussion. I dont need a full year forecast and Im only looking for weekly forecasts. I only used one year ahead as a test data set. I should mention that not all the time, integrating the observed data for January 2010 to the historic data will improve forecasts for Februray. I did an experiment and compared the errors for februray forecast without adding new observed data and February forecast by including January data. Surprisingly, it makes my predictions worse. This statement does not work for all data sets as I tested. $\endgroup$
    – user12
    Commented Sep 30, 2014 at 12:21
2
$\begingroup$

You data https://www.autobox.com/FATEME/TestData.xlsx is here . I suggest other readers take the data and try to analyze it. On first view the 1096 historical daily values appear to have a large number of "outliers". These unusual values are in effect mostly usual as certain holidays have a significant effect. The problem you are having has to with your design matrix. You assume that the effect is day-of-the-week vs weekend. This is wrong because each day has it's own effect and certainly the 5 work days are not equal and the two weekend days are not equal in their importance. There have been major shifts in the days-of-the=week effects. Furthermore specific days-of-the-month and specific months of the year are very important in addition to Friday before Holidays impacts. Next there is a visually obvious level shift ( on or about 1/27/09 ) in your data that needs to be accounted for such that the final model coefficients are robust and meaningful.

Using AUTOBOX (a piece of software available from http://www.autobox.com/cms/ which I have helped develop) in a totally automatic fashion a model containing both deterministic effects and memory (ARIMA) was forged.

Attached is the equation (in two parts)enter image description here and enter image description here. A test for the sufficiency of a model is the ACF of the errors suggests randomness. Following is the Forecast plot.enter image description here. I have added a representativeenter image description here screen shot of the forecasts presented in tabular fashion (forecasts then lower limit then upper limit).

With apologies in advance the 1096 values were able to be characterized with 58 statistically significant coefficients.

The summary statistics from the model are as follows enter image description here

EDIT: I have added the 366 forecasts from 1 origin 12/31/2009 https://www.autobox.com/FATEME/AB50PRO.123

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ That blue background screenshot is very difficult to read (at least for me it kind hurts the eyes). Another advice is to make your code's output pictures clickable so one can increase and visualize them with a greater size. If you are interested, here is the post which explains how to do it. $\endgroup$
    – Andre Silva
    Commented Sep 28, 2014 at 14:46
  • 1
    $\begingroup$ This is impressive (+1). Since you have challenged others to analyze these data you need to report the performance of your forecasts on the test dataset, which has been supplied in the spreadsheet specifically to evaluate competing models. I suspect an approach that extends your technology to generalized linear models ought to work even better, because these data could fruitfully be conceived of as realizations of a Poisson process (in which the underlying mean process is of the form you software currently handles). Do you test the homoscedasticity of the residuals? $\endgroup$
    – whuber ♦
    Commented Sep 28, 2014 at 15:50
  • $\begingroup$ My problem is that measuring forecast accuracy for the next 366 values from 1 origin is a sample of 1 . I $\endgroup$
    – IrishStat
    Commented Sep 28, 2014 at 16:26
  • $\begingroup$ My problem is that measuring forecast accuracy for the next 366 values from 1 and only 1 origin is a sample of 1. If one brought in the next value (1/1/2010) and obtained a new set of forecasts for the remaining 365 values a second/different estimate of accuracy could be computed etc..... In general one has to specify the number of values to be used to measure accuracy and then do so from many origins. It is very naive to measure accuracy for daily data for a holdout period of 366 days. I would suggest a 30 day out holdout from 336 origins could be appropriate BUT a lot of work for this task. $\endgroup$
    – IrishStat
    Commented Sep 28, 2014 at 16:38
  • 1
    $\begingroup$ It's going too far to accuse the OP of naiveté when you do not know their objectives, especially the time horizon they must use for predictions. It would be fair, though, for you to posit specific purposes or contexts and then propose (and apply) reasonable methods of using the held-out data for assessing the accuracy of forecasts. $\endgroup$
    – whuber ♦
    Commented Sep 29, 2014 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.