ARIMA model updating for manufacturing machine When should one use the different types of ARIMA model as mentioned below:


*

*Estimate the model order in the training data set and use the same order to forecast future values (updating the parameter estimates)

*Use a rolling window (e.g. 30 day) to make a new forecast by estimating model order and parameters everytime to make a new forecast.


I tried the above strategies to forecast the daily downtimes of the machine as a percentage of scheduled time in manufacturing. I used the cross-validation methods (as mentioned by Rob Hyndman) and found that the rolling window with the new estimation of model order (p,d,q) and parameters every time a new forecast is a better forecasting model (MAPE for the second model is 3.34% compared to 10.25% for the first model)
I got confused on understanding why the model performs better with the method 2 or Am I doing something wrong? How do we explain this model in this manufacturing case? The breakdown pattern of the machine changes when repair work is done every time as breakdowns also depend on the "quality" of the repair work. So it is hard to fix model order p,d,q for the machine. Could this be a reason for the model 2 to perform better? any thoughts? Also, is it practical and advisable to estimate the model order everytime ?
This post explains the answer partially but still, couldn't understand the context with my data set and application.
 A: After receipt of your data (NOB= 315 daily values) and some reflection on your problem. When faced with NOB observations one can build a model incorporating ARIMA structure , level shifts , trend structure , outlier/pulse structure. Sound analysis can detect structural breaks either in the mean or in the error variance or in the ARMA component essentially answering the question about the homogeneity of the data. Posible solutions after detecting change in model parameters and/ or change in error variance is to segment the data and reprocess with the most recent set of values.
As I understand what you want to do is to determine that window ( # of observations used to form a reasonable model ) based upon a "1 period" out of sample MAPE. Now the smaller the window the more questionable is the identified model while the larger the window (given proven homogeneity) the more reliable the model. This was found to be true for your data.
What I did was to construct a computer-based automatic study to determine that window size which provide the best mape. My initial test results is that the larger the window the smaller the average mape suggesting a window size of about 200 values as optimal. This size window (200) yielded a mape of 7.5% while a smaller window (40) averaged 8.5% .  It is widely known that generally speaking the more observations that you have to construct a model (to a point !) the better the statistical identification results given that you have rectified any gaussian violations.
I will now detail the experiment . Take the first 200 values and simultaneously identify an ARIMA component and any necessary deterministic structure ( pulses,step/level shifts, local time trends) while testing for parameter change points and error variance change points yielding a forecast and a mape for the 201st data point. I then selected observations 2-201 and totally reran the analysis yielding a mape for the 202nd data point. I proceeded to do this until the last analysis using observations 115 to 314 to predict the 315th data point. In total we now have 115 estimates of the 1 period out mape and can then average them to get a representative value.
This whole experiment was redone using window sizes of 250, 200 ,150, 100 and 40 
with corresponding average mapes of 7.6 , 7.5 , 7.8 , 8.2 and 8.5 . 
The overall winner was "200" with "40" a clear loser . The AUTOBOX procedure was used to form the model in a totally automatic and transparent fashion and was accomplished in a few minutes. This is the basic info ( and partial flow chart ) for how AUTOBOX works http://www.autobox.com/cms/index.php/afs-university/autobox-examples/modeling-with-autobox/ . In reflection I have been personally responsible for some of the development of AUTOBOX and can vouch for it's professional and thorough approach to forming models.
In closing I am not sure why one is trying to assess the minimum # of historical values to be used (window size) unless there is some sort of data storage capacity issue.
AUTOBOX is fully conformable to evaluating alternative modelling settings/switches . I set the following standard settings here as:
a) enable Intervention Detection
b) enable parameter change detection
c) enable variance change detection
All combinations of these switches is 8 . By altering the "switches" one can then estimate a mape premised on the switch settings. It is then possible to select which modelling options would be optimal for each machine. There a few other useful settings available like disabling model stepdown .
