8
$\begingroup$

i just finished developing an ARMAX model with python (mostly statsmodels) in order to forecast some data. My next step is to test the data (24 time series) with the given ARMAX model. As i need to write a proper academic documentation about all tests i use and the way i test my data, i need to have a proper testing design.

I found some good designs here: http://www.autobox.com/cms/index.php/blog/entry/build-or-make-your-own-arima-forecasting-model

However, my model and testing design looks like this:

  1. Data preparation (identification and Difference data to obtain stationary series)
    • Descriptive statistics for each hour (count, mean, skewness etc)
    • Augmented Dickey Fuller Test to detect stationarity of given time series

--> excel-documentation: Stationarity of time series exists!

  1. Model Selection (Examine data, ACF, PACF to identify potential (choosing tentative p and q)
    • Plot and analyse ACF and PACF
    • Automatic selection of lowest Information criterion (AIC, BIC, HQIC)

--> excel-documentation: ACF and PACF plot/picture, interpretation of plot, Lowest information criterion (AIC, BIC, HQIC)

  1. Estimation (Estimate parameters in potential model and testing. Select best model using suitable criterion Diagnostic)
    • choose p- and q-parameter according to lowest AIC'

--> excel-documentation: which parameters are going to be used for arma.prediction

  1. Diagnostic (falsification of model selection process)
    • Durbin-Watson Test to detect presence of autocorrelation
    • plot residuals to see structure i.e. white noise
    • Normality test (D'Angelo and Pearson) to see difference from normal distribution
    • qqplot of the residuals against quantiles of t-distribution (in addition to normality test)
    • plot ACF and PACF of residuals to detect white noise
    • Ljung-Box Test to test overall randomness based on a number of lags

--> excel-documentation: Durbin-Watson-Test-Results, Normality-Test-Results, Summary of Ljung-Box-Test (Q>0, y/n?)

  1. Forecasting (use model to forecast)
    • run model
    • analyse arma.summary-table
    • compare predicted value with real value (in-sample analysis)

--> excel-documentation: prediction value for given p- and q-values (see. '3. Estimation')

  1. Verification (Mean absolute percentage error (MAPE) for in-sample analysis)
    • compare predicted value with real value

--> excel-documentation: MAPE for given p- and q-values

  • go back to '3. Estimation' and run again if Diagnostic-results and MAPE are not satisfactory

  • Maximum Re-Running-Time based on optimal selection of information criterion: if model output is not satisfactory, choose higher and lower p- and q-values. Use lowest BIC and/or HQIC (if AIC, BIC and HQIC suggest same p- and q values, use different approach)

Would be great if someone can take a minute and tell me if this sounds legitimate from a academic point of view.

Tanks in advance

Improve this question
$\endgroup$
6
  • $\begingroup$ What's academic point of view? Are you writing a paper? $\endgroup$
    – Aksakal
    Jan 5, 2015 at 16:15
  • 3
    $\begingroup$ Looks good to me, but it really depends on the purpose of the academic paper. In some sense, Box-Jenkins methodology is old news; 1970s stuff! Most readers would, I imagine, be more interested in the results of your research rather than seeing into the kitchen. It's not unusual to read something like "by applying the Box-Jenkins methodology an ARMA(2,1) model was chosen" without going into the actual details. This would certainly be the case in economics where a time-series model would only be used as a benchmark model. Comparing forecast performance w/ rival models more interesting probably? $\endgroup$
    – Graeme Walsh
    Jan 5, 2015 at 16:17
  • 2
    $\begingroup$ @ Aksakal: I am writing my Master-Thesis $\endgroup$
    – Peter Knutsen
    Jan 5, 2015 at 16:50
  • $\begingroup$ OK. Master-Thesis is a different story. Looks like you're demonstrating that you know what you're doing, but it might be worth considering points made here: stats.stackexchange.com/questions/131128/… Some is at odds to what you done - regarding AIC in step 3? $\endgroup$
    – Graeme Walsh
    Jan 5, 2015 at 17:03
  • $\begingroup$ @Graeme Walsh: Thanks for your feedback. I know, the model is a bit outdated. But python only offers classical approaches (i.e. ARIMA-models). My task is to build a forecasting model with python and dont have enough programming experience to build a model (e.g. neural-network) on my own. So i stick with ARMA from the 70s. Plus, i red some papers suggesting that ARMA models (somehow modified) show good or even better results then new approaches. I think it would be great to compare my final results with R or MATLAB to show that python delivers proper ARMA predictions (scientific value!?). $\endgroup$
    – Peter Knutsen
    Jan 5, 2015 at 17:11

1 Answer 1

Reset to default
3
$\begingroup$

"Data preparation (identification and Difference data to obtain stationary series)" . Non-stationarity may be the symptom while the cause may be a simple change in the mean or a simple change in trend or a simple change in parameters or a simple change in error variance. Alternatively/conversely an unusual value (pulse) will increase the variance and increase the covariance thus the acf will be downwards biased yielding possibly false conclusions about non-existent ARIMA structure. Either way your design does not understand/follow the flow charts presented in your reference.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer! As far as i know, there are no simple changes which might cause non-stationarity in my time series. The only thing i do to modify my TS is to exclude some days (i.e. weekends and holidays). So, testing for stationarity should be adequate, dont you think? I also might add some data smoothing to avoid unusual values (price-spikes). The refference i use for my model is the following link (The flow chart presented in my question can be seen as an orientation rather then a reference.) $\endgroup$
    – Peter Knutsen
    Jan 5, 2015 at 18:25
  • $\begingroup$ If you have daily data you might want to think about a design that tests for and includes day-of-the-week series, holiday indicators ( lead and lag) , long-weekend effects , level shift in the mean , change points in day-of-the-week effects. To paraphrase a colleague of mine " nearly all complicated problems have simple solutions which when tried don't work too well , if at all ". While your approach should be simple it shouldn't be too simple. $\endgroup$
    – IrishStat
    Jan 5, 2015 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.