Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I would like to build an algorithm that would be able to analyze any time series and "automatically" choose the best traditional/statiscal forecasting method (and its parameters) for the analyzed time series data.
Would it be possible to do something like this? If yes, can you give me some tips on how can this be approached?
First you need to note that the approach outlined by IrishStat is specific to ARIMA models, not to any generic set of models.
To answer your main question "Is it possible to automate time series forecasting?":
Yes it is. In my field of demand forecasting, most commercial forecasting packages do so. Several open source packages do so as well, most notably Rob Hyndman's auto.arima() (automated ARIMA forecasting) and ETS() (automated exponential smoothing forecasting) functions from the open source Forecast package in R see here for details on these two functions. There's also a Python implementation of auto.arima called Pyramid, although in my experience it is not as mature as the R packages.
Both the commercial products that I mentioned and the open source packages I mentioned work based on the idea of using information criteria to choose the best forecast: You fit a bunch of models, and then select the model with the lowest AIC, BIC, AICc, etc....(typically this is done in lieu of out of sample validation).
There is however a major caveat: all of these methods work within a single family of models. They choose the best possible model amongst a set of ARIMA models, or the best possible model amongst a set of exponential smoothing models.
It is much more challenging to do so if you want to choose from different families of models, for example if you want to choose the best model from ARIMA, Exponential smoothing and the Theta method. In theory, you can do so in the same way that you do within a single family of models, i.e. by using information criteria. However in practice, you need to calculate the AIC or BIC in exactly the same way for all models considered, and that is a significant challenge. It might be better to use time series cross-validation, or out of sample validation instead of information criteria, but that will be much more computationally intensive (and tedious to code).
Facebook's Prophet package also automates forecast generation based on General Additive Models See here for details. However Prophet fits only one single model, albeit a very flexible model with many parameters. Prophet's implicit assumption is that a GAM is "the one model to rule them all", which might not be theoretically justified but is very pragmatic and useful for real world scenarios.
Another caveat that applies to all of the above mentioned methods: Presumably you want to do automated time series forecasting because you want to forecast multiple time series, too many to analyze manually. Otherwise you could just do your own experiments and find the best model on your own. You need to keep in mind that an automated forecasting approach is never going to find the best model for each and every time series - it is going to give a reasonably good model on average over all the time series, but it is still possible that some of those time series will have better models than the ones selected by the automated method. See this post for an example of this. To put it simply, if you are going to go with automated forecasting - you will have to tolerate "good enough" forecasts instead of the best possible forecasts for each time series.
My suggested approach encompasses models that are much more general than ARIMA as they include the potential for seasonal dummies that may change over time , multiple levels ,multiple trends , parameters that may change over time and even error variances that may change over time. This family is more precisely called ARMAX models but for complete transparency does exclude a (rare) variant that has multiplicative structure.
You asked for tips and I believe that this might be a good one to get you started.
I would suggest that you write code to follow/emulate this flowchart/workflow. The "best model" could be found by evaluating the criterion that you specify ... it could be the MSE/AIC of the fitted data or it could be the MAPE/SMAPE of withheld data or any criterion of your choice.
Be aware as the detailing of each of these steps can be quite simple if you are unaware of some of the specific requirements/objectives/constraints of time series analysis BUT it can be (should be !) more complex if you have a deeper understanding/learning/appreciation of the complexities/opportunities present in thorough time series analysis.
I have been asked to provide further direction as to how one should go about automating time series modelling ( or modelling in general ) https://stats.stackexchange.com/search?q=peeling+an+onion contains some of my guidance on "peeling onions" and related tasks .
AUTOBOX actually details and shows the interim steps as it forms a useful model and could be a useful teacher in this regard. The whole scientific idea is to "add what appears to be needed" and "delete what appears to be less than useful" . This is the iterative process suggested by Box and Bacon in earlier times.
Models need to be complex enough (fancy enough) but not too complex (fancy). Assuming that simple methods work with complex problems is not consistent with scientific method following Roger Bacon and tons of followers of Bacon. As Roger Bacon once said and I have often paraphrased : To do science is to search for repeated patterns. To detect anomalies is to identify values that do not follow repeated patterns. For whoever knows the ways of Nature will more easily notice her deviations and, on the other hand, whoever knows her deviations will more accurately describe her ways. One learns the rules by observing when the current rules fail.In the spirt pf Bacon by identifying when the currently identified "best model/theory" is inadeqaute one can then iterate to "a better representation"
In my words "Tukey proposed Exploratory Data Analysis (EDA) which suggested schemes of model refinement based upon evidented model deficiency suggested by the data" . This is the heart of AUTOBOX and of science. EDA is for seeing what the data can tell us beyond the formal modeling or hypothesis testing task.
The litmus test of an automatic modelling program is quite simple . Does it separate signal and noise without over-fitting ? Empirical evidence suggests that this can and has been done. Forecasting accuracies are often misleading because the future is not accountable for the past and depending on which origin you pick results can and do vary.
Short Answer
While it could be possible to do something like this, in many cases you are probably better off forecasting time series using a more manual approach.
Long Answer
The approach you describe is similar to what is seen in the machine learning community where a tremendous amount of focus is put on model selection and parameter estimation. For example, there are countless papers on how to optimize a neural net to attain strong results on the ImageNet dataset. Part of the reason there is such an emphasis on this, is that in the research process it is important to compare your model to other models on benchmark datasets. To make sure your results are comparable to others' reported results you cannot manually bring in outside information to improve your model's performance.
While this automatic-forecasting approach is not wrong per se, in the time series setting often the most important step in an analysis is bringing in exogenous variables to help explain a time series. For example, if one is attempting to forecast a time series of average house prices in New York, what would matter most in forecasting is determining what factors influence house prices (e.g. population growth, interest rates, unemployment) and then coming up with a reasonable forecast of these variables to inform your forecast of house prices. The time dependence of the residuals (which is what most traditional statistical methods attempt to model), while still important to consider in the model, would be likely be far less important in improving a forecast than the selection and forecasting of the aforementioned exogenous variables.
